2024-01-18T07:06:37.329 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-18T07:06:37.330 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659 --name yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/orig.config.yaml 2024-01-18T07:06:37.377 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T07:06:37.474 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659 branch: wip-yuri10-testing-2024-01-17-0759-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7519659' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-01-17-0759-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-01-17-0759-pacific sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2168 sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-01-17-0759-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlGicQZ8bRPfx7Psd5HuE7IMf7D2XHgU74GViHlSAf+oFgpasZyB7V1umOdDbm/UO3j9E8MVISyJbkkSrrLx+c= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwbKpEW+Zr3p+nSrCGIYNgaAS97TE1Zn82PniN69HLG85jkDjGrf1sxJcqwdu6DbvlduVBHq8PTELL3S0uwYD8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-17_22:43:41 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-18T07:06:37.474 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa; will attempt to use it 2024-01-18T07:06:37.475 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks 2024-01-18T07:06:37.475 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-18T07:06:37.476 INFO:teuthology.task.internal:Checking packages... 2024-01-18T07:06:37.499 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0' 2024-01-18T07:06:37.499 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-18T07:06:37.499 INFO:teuthology.packaging:ref: None 2024-01-18T07:06:37.499 INFO:teuthology.packaging:tag: None 2024-01-18T07:06:37.499 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-01-17-0759-pacific 2024-01-18T07:06:37.499 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:06:37.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-01-17-0759-pacific 2024-01-18T07:06:37.725 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-454.g491b5e13 2024-01-18T07:06:37.727 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-18T07:06:37.734 INFO:teuthology.task.internal:no buildpackages task found 2024-01-18T07:06:37.734 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-18T07:06:37.751 INFO:teuthology.task.internal:Saving configuration 2024-01-18T07:06:37.768 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-18T07:06:37.774 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T07:06:37.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 07:01:32.459685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlGicQZ8bRPfx7Psd5HuE7IMf7D2XHgU74GViHlSAf+oFgpasZyB7V1umOdDbm/UO3j9E8MVISyJbkkSrrLx+c='} 2024-01-18T07:06:37.818 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 07:01:32.461427', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwbKpEW+Zr3p+nSrCGIYNgaAS97TE1Zn82PniN69HLG85jkDjGrf1sxJcqwdu6DbvlduVBHq8PTELL3S0uwYD8='} 2024-01-18T07:06:37.818 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-18T07:06:37.825 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-18T07:06:37.825 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-18T07:06:37.825 INFO:teuthology.run_tasks:Running task console_log... 2024-01-18T07:06:37.886 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7035f70ee0>, signals=[15]) 2024-01-18T07:06:37.886 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-18T07:06:37.892 INFO:teuthology.task.internal:Opening connections... 2024-01-18T07:06:37.893 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-01-18T07:06:37.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:06:37.966 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-01-18T07:06:37.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:06:38.038 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-18T07:06:38.045 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-01-18T07:06:38.066 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-01-18T07:06:38.067 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-01-18T07:06:38.126 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-18T07:06:38.127 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-18T07:06:38.128 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-01-18T07:06:38.152 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-01-18T07:06:38.173 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-01-18T07:06:38.173 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-01-18T07:06:38.233 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-01-18T07:06:38.233 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="8" 2024-01-18T07:06:38.233 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-01-18T07:06:38.233 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="8" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el8" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-18T07:06:38.234 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-18T07:06:38.235 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-01-18T07:06:38.252 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-18T07:06:38.260 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-18T07:06:38.359 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-18T07:06:38.359 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-01-18T07:06:38.361 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-01-18T07:06:38.382 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-18T07:06:38.387 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-18T07:06:38.388 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-01-18T07:06:38.422 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-01-18T07:06:38.459 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-18T07:06:38.511 INFO:teuthology.run_tasks:Running task kernel... 2024-01-18T07:06:38.526 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-18T07:06:38.526 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-18T07:06:38.527 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-18T07:06:38.527 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-18T07:06:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:06:38.546 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-01-18T07:06:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:06:38.548 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-01-18T07:06:38.606 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-489.el8.x86_64 2024-01-18T07:06:38.606 INFO:teuthology.task.kernel:Running kernel on smithi084: 4.18.0-489.el8.x86_64 2024-01-18T07:06:38.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-01-18T07:06:38.610 INFO:teuthology.orchestra.run.smithi106.stdout:4.18.0-489.el8.x86_64 2024-01-18T07:06:38.611 INFO:teuthology.task.kernel:Running kernel on smithi106: 4.18.0-489.el8.x86_64 2024-01-18T07:06:38.611 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum install -y kernel 2024-01-18T07:06:41.581 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 49 MB/s | 26 MB 00:00 2024-01-18T07:06:42.009 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - AppStream 35 MB/s | 26 MB 00:00 2024-01-18T07:06:46.963 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-01-18T07:06:47.359 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 10 MB 00:00 2024-01-18T07:06:48.963 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2024-01-18T07:06:49.334 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-01-18T07:06:49.430 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-18T07:06:49.666 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-18T07:06:49.785 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-01-18T07:06:50.012 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-01-18T07:06:51.141 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-01-18T07:06:54.079 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2024-01-18T07:06:54.916 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 426 kB/s | 24 kB 00:00 2024-01-18T07:06:57.821 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-18T07:06:57.821 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-18T07:06:57.871 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 425 kB/s | 24 kB 00:00 2024-01-18T07:06:57.929 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:06:57.930 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout:Install 3 Packages 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 89 M 2024-01-18T07:06:57.931 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 96 M 2024-01-18T07:06:57.932 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:06:58.500 INFO:teuthology.orchestra.run.smithi084.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 19 MB/s | 10 MB 00:00 2024-01-18T07:06:59.517 INFO:teuthology.orchestra.run.smithi084.stdout:(2/3): kernel-core-4.18.0-535.el8.x86_64.rpm 28 MB/s | 43 MB 00:01 2024-01-18T07:06:59.859 INFO:teuthology.orchestra.run.smithi084.stdout:(3/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 19 MB/s | 36 MB 00:01 2024-01-18T07:06:59.859 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:06:59.859 INFO:teuthology.orchestra.run.smithi084.stdout:Total 46 MB/s | 89 MB 00:01 2024-01-18T07:07:00.544 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:07:00.768 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-18T07:07:00.768 INFO:teuthology.orchestra.run.smithi106.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-18T07:07:00.876 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:07:00.876 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout:Install 3 Packages 2024-01-18T07:07:00.877 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:00.878 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 89 M 2024-01-18T07:07:00.878 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 96 M 2024-01-18T07:07:00.878 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:07:00.945 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:07:00.945 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:07:01.831 INFO:teuthology.orchestra.run.smithi106.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-01-18T07:07:02.182 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:07:02.203 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:07:02.897 INFO:teuthology.orchestra.run.smithi106.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 20 MB/s | 36 MB 00:01 2024-01-18T07:07:03.356 INFO:teuthology.orchestra.run.smithi106.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 19 MB/s | 43 MB 00:02 2024-01-18T07:07:03.356 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:07:03.356 INFO:teuthology.orchestra.run.smithi106.stdout:Total 36 MB/s | 89 MB 00:02 2024-01-18T07:07:04.033 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:07:04.454 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:07:04.455 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:07:04.953 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:07:05.477 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:05.699 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:07:05.727 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:07:08.517 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:07:08.969 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:09.053 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:09.476 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:12.536 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:12.896 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:12.943 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:13.256 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:16.365 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:16.651 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:35.181 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:35.205 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:35.534 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:35.534 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:35.534 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:07:36.027 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:07:36.209 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T07:07:36.423 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T07:07:36.423 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T07:07:36.424 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-18T07:07:36.424 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-18T07:07:36.424 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-18T07:07:36.424 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-18T07:07:36.424 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-01-18T07:07:36.424 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T07:07:36.634 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T07:07:36.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T07:07:36.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-01-18T07:07:36.667 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-01-18T07:07:36.667 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-01-18T07:07:36.667 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.02 2024-01-18T07:07:36.667 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 129.el8 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Group : System Environment/Base 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 9265586 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Relocations : (not relocatable) 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : CentOS BuildSystem 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-01-18T07:07:36.668 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:07:36.669 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-01-18T07:07:36.670 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-18T07:07:36.670 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 4.18.0-535.el8.x86_64 2024-01-18T07:07:36.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-18T07:07:38.368 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:38.369 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-01-18T07:07:38.392 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:38.741 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:38.741 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-18T07:07:38.741 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:39.223 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:07:39.352 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T07:07:39.587 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T07:07:39.588 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T07:07:39.588 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-18T07:07:39.588 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-18T07:07:39.588 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-18T07:07:39.588 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-18T07:07:39.588 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi106.front.sepia.ceph.com, path=None, version=distro) 2024-01-18T07:07:39.588 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T07:07:39.799 INFO:teuthology.orchestra.run.smithi106.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-18T07:07:39.799 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-18T07:07:39.799 DEBUG:teuthology.orchestra.run.smithi106:> sudo rpm -qi grub2-tools 2024-01-18T07:07:39.832 INFO:teuthology.orchestra.run.smithi106.stdout:Name : grub2-tools 2024-01-18T07:07:39.832 INFO:teuthology.orchestra.run.smithi106.stdout:Epoch : 1 2024-01-18T07:07:39.832 INFO:teuthology.orchestra.run.smithi106.stdout:Version : 2.02 2024-01-18T07:07:39.832 INFO:teuthology.orchestra.run.smithi106.stdout:Release : 129.el8 2024-01-18T07:07:39.832 INFO:teuthology.orchestra.run.smithi106.stdout:Architecture: x86_64 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Group : System Environment/Base 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Size : 9265586 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:License : GPLv3+ 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Relocations : (not relocatable) 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Packager : CentOS BuildSystem 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Vendor : CentOS 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Summary : Support tools for GRUB. 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout:Description : 2024-01-18T07:07:39.833 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:hardware devices. 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:07:39.834 INFO:teuthology.orchestra.run.smithi106.stdout:This subpackage provides tools for support of all platforms. 2024-01-18T07:07:39.835 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-18T07:07:39.835 INFO:teuthology.task.kernel:Updating grub on smithi106 to boot 4.18.0-535.el8.x86_64 2024-01-18T07:07:39.835 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-18T07:07:40.629 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-01-18T07:07:40.631 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-01-18T07:07:40.665 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.reRUutCXRu 2024-01-18T07:07:40.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.reRUutCXRu 2024-01-18T07:07:40.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.reRUutCXRu 2024-01-18T07:07:40.882 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.reRUutCXRu is 6KB 2024-01-18T07:07:40.930 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.reRUutCXRu 2024-01-18T07:07:40.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-01-18T07:07:41.023 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-18T07:07:41.023 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-18T07:07:41.023 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-18T07:07:41.023 INFO:teuthology.orchestra.run.smithi084.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-18T07:07:41.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-18T07:07:41.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-01-18T07:07:41.544 INFO:teuthology.orchestra.run.smithi106.stderr:Generating grub configuration file ... 2024-01-18T07:07:43.926 INFO:teuthology.orchestra.run.smithi106.stderr:done 2024-01-18T07:07:43.928 DEBUG:teuthology.orchestra.run.smithi106:> mktemp 2024-01-18T07:07:43.960 INFO:teuthology.orchestra.run.smithi106.stdout:/tmp/tmp.bYR1RlupPr 2024-01-18T07:07:43.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bYR1RlupPr 2024-01-18T07:07:44.033 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0666 /tmp/tmp.bYR1RlupPr 2024-01-18T07:07:44.173 DEBUG:teuthology.orchestra.remote:smithi106:/tmp/tmp.bYR1RlupPr is 6KB 2024-01-18T07:07:44.220 DEBUG:teuthology.orchestra.run.smithi106:> rm -fr /tmp/tmp.bYR1RlupPr 2024-01-18T07:07:44.238 DEBUG:teuthology.orchestra.run.smithi106:> sudo /bin/ls /boot/loader/entries || true 2024-01-18T07:07:44.316 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-18T07:07:44.317 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-18T07:07:44.317 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-18T07:07:44.317 INFO:teuthology.orchestra.run.smithi106.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-18T07:07:44.318 DEBUG:teuthology.orchestra.run.smithi106:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-18T07:07:44.494 DEBUG:teuthology.orchestra.run.smithi106:> sudo shutdown -r now 2024-01-18T07:08:11.220 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-18T07:08:11.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:08:11.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:08:14.501 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-18T07:08:14.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-18T07:08:14.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:08:29.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-01-18T07:08:38.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:08:38.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:08:41.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-01-18T07:08:53.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:08:53.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:09:12.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-01-18T07:09:14.505 DEBUG:teuthology.orchestra.remote:timed out 2024-01-18T07:09:23.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-18T07:09:23.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:09:23.906 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-01-18T07:09:24.233 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-18T07:09:24.234 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-18T07:09:24.234 DEBUG:teuthology.orchestra.run.smithi106:> uname -r 2024-01-18T07:09:24.251 INFO:teuthology.orchestra.run.smithi106.stdout:4.18.0-535.el8.x86_64 2024-01-18T07:09:24.251 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-18T07:09:24.251 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-18T07:09:24.251 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-18T07:09:25.252 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-18T07:09:25.252 DEBUG:teuthology.orchestra.run.smithi106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-18T07:09:25.323 INFO:teuthology.orchestra.run.smithi106.stdout:ttyS1 2024-01-18T07:09:25.336 DEBUG:teuthology.parallel:result is None 2024-01-18T07:09:27.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:09:27.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:09:27.699 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-01-18T07:09:28.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:09:28.018 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-18T07:09:28.018 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-01-18T07:09:28.034 INFO:teuthology.orchestra.run.smithi084.stdout:4.18.0-535.el8.x86_64 2024-01-18T07:09:28.035 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-18T07:09:28.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-18T07:09:28.035 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-18T07:09:29.035 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-18T07:09:29.035 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-18T07:09:29.703 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-01-18T07:09:29.750 DEBUG:teuthology.parallel:result is None 2024-01-18T07:09:29.750 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-18T07:09:29.758 INFO:teuthology.task.internal:Creating test directory... 2024-01-18T07:09:29.758 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-18T07:09:29.816 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-18T07:09:30.222 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-18T07:09:30.232 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-18T07:09:30.243 INFO:teuthology.task.internal:Creating archive directory... 2024-01-18T07:09:30.243 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-18T07:09:30.286 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-18T07:09:30.394 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-18T07:09:30.401 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-18T07:09:30.401 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-18T07:09:30.404 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-18T07:09:30.439 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T07:09:30.453 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T07:09:30.458 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T07:09:30.471 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T07:09:30.472 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-18T07:09:30.479 INFO:teuthology.task.internal:Configuring sudo... 2024-01-18T07:09:30.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-18T07:09:30.481 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-18T07:09:30.523 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-18T07:09:30.532 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-18T07:09:30.532 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-18T07:09:30.549 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-18T07:09:30.582 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T07:09:30.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T07:09:30.723 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T07:09:30.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T07:09:30.886 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:09:30.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-18T07:09:30.953 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T07:09:30.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T07:09:31.056 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T07:09:31.138 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T07:09:31.183 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:09:31.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-18T07:09:31.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-01-18T07:09:31.252 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-01-18T07:09:31.298 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T07:09:31.352 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T07:09:31.673 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-18T07:09:31.683 INFO:teuthology.task.internal:Starting timer... 2024-01-18T07:09:31.683 INFO:teuthology.run_tasks:Running task pcp... 2024-01-18T07:09:31.693 INFO:teuthology.run_tasks:Running task selinux... 2024-01-18T07:09:31.700 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-18T07:09:31.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-01-18T07:09:31.772 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-01-18T07:09:31.774 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-01-18T07:09:31.844 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-01-18T07:09:31.846 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-18T07:09:31.846 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-01-18T07:09:31.867 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-01-18T07:09:31.867 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-01-18T07:09:31.900 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-01-18T07:09:31.900 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi106.front.sepia.ceph.com': 'permissive'} 2024-01-18T07:09:31.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T07:09:31.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:09:31.933 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T07:09:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:09:31.968 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-18T07:09:31.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-01-18T07:09:31.999 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-01-18T07:09:32.033 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-18T07:09:32.043 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-18T07:09:32.049 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-18T07:09:32.157 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-18T07:09:32.174 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-18T07:09:32.175 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi106.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-18T07:15:04.222 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com')] 2024-01-18T07:15:04.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:15:04.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:15:04.300 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-01-18T07:15:04.366 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-01-18T07:15:04.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-18T07:15:04.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T07:15:04.441 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-01-18T07:15:04.511 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-01-18T07:15:04.511 INFO:teuthology.run_tasks:Running task clock... 2024-01-18T07:15:04.524 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-18T07:15:04.524 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-18T07:15:04.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T07:15:04.527 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-18T07:15:04.527 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T07:15:04.556 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-18T07:15:04.570 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-18T07:15:04.582 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-18T07:15:04.596 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-18T07:15:04.634 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-01-18T07:15:04.645 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-01-18T07:15:04.659 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-18T07:15:04.672 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-18T07:15:04.672 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-01-18T07:15:04.683 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-01-18T07:15:04.696 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-18T07:15:04.709 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-18T07:15:04.744 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.747 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.774 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-01-18T07:15:04.777 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T07:15:04.777 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-01-18T07:15:04.777 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.778 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.778 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.778 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:15:04.779 INFO:teuthology.run_tasks:Running task pexec... 2024-01-18T07:15:04.788 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-18T07:15:04.789 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-01-18T07:15:04.789 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T07:15:04.789 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi106.front.sepia.ceph.com 2024-01-18T07:15:04.789 DEBUG:teuthology.orchestra.run.smithi106:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T07:15:05.535 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:21 ago on Thu 18 Jan 2024 07:13:44 AM UTC. 2024-01-18T07:15:05.568 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:21 ago on Thu 18 Jan 2024 07:13:44 AM UTC. 2024-01-18T07:15:07.464 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:15:07.465 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:07.465 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:15:07.465 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout:Resetting modules: 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:07.466 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:07.514 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout:Resetting modules: 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools 2024-01-18T07:15:07.515 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:07.516 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:15:07.516 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:07.516 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:07.728 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:15:07.791 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:15:08.547 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:24 ago on Thu 18 Jan 2024 07:13:44 AM UTC. 2024-01-18T07:15:08.587 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:24 ago on Thu 18 Jan 2024 07:13:44 AM UTC. 2024-01-18T07:15:10.464 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout:Installing group/module packages: 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-18T07:15:10.499 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout:Installing module profiles: 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools/common 2024-01-18T07:15:10.500 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling module streams: 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout: container-tools rhel8 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-01-18T07:15:10.501 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:10.504 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 27 M 2024-01-18T07:15:10.504 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 89 M 2024-01-18T07:15:10.504 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:15:10.506 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:15:10.541 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:10.541 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:15:10.541 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:10.541 INFO:teuthology.orchestra.run.smithi106.stdout:Installing group/module packages: 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-18T07:15:10.542 INFO:teuthology.orchestra.run.smithi106.stdout:Installing module profiles: 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools/common 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout:Enabling module streams: 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout: container-tools rhel8 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout:Install 13 Packages 2024-01-18T07:15:10.543 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:10.544 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 27 M 2024-01-18T07:15:10.544 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 89 M 2024-01-18T07:15:10.544 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:15:10.998 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 6.9 MB/s | 750 kB 00:00 2024-01-18T07:15:11.002 INFO:teuthology.orchestra.run.smithi106.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 7.3 MB/s | 750 kB 00:00 2024-01-18T07:15:11.032 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 7.2 MB/s | 1.0 MB 00:00 2024-01-18T07:15:11.035 INFO:teuthology.orchestra.run.smithi106.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 7.5 MB/s | 1.0 MB 00:00 2024-01-18T07:15:11.057 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 644 kB/s | 38 kB 00:00 2024-01-18T07:15:11.061 INFO:teuthology.orchestra.run.smithi106.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2024-01-18T07:15:11.083 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.7 MB/s | 241 kB 00:00 2024-01-18T07:15:11.086 INFO:teuthology.orchestra.run.smithi106.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.7 MB/s | 241 kB 00:00 2024-01-18T07:15:11.116 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.2 MB/s | 178 kB 00:00 2024-01-18T07:15:11.120 INFO:teuthology.orchestra.run.smithi106.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.2 MB/s | 178 kB 00:00 2024-01-18T07:15:11.146 INFO:teuthology.orchestra.run.smithi106.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 981 kB/s | 25 kB 00:00 2024-01-18T07:15:11.150 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 745 kB/s | 25 kB 00:00 2024-01-18T07:15:11.209 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 26 MB/s | 3.9 MB 00:00 2024-01-18T07:15:11.212 INFO:teuthology.orchestra.run.smithi106.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 26 MB/s | 3.9 MB 00:00 2024-01-18T07:15:11.237 INFO:teuthology.orchestra.run.smithi106.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-01-18T07:15:11.252 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 918 kB/s | 94 kB 00:00 2024-01-18T07:15:11.376 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 20 MB/s | 9.6 MB 00:00 2024-01-18T07:15:11.379 INFO:teuthology.orchestra.run.smithi106.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 20 MB/s | 9.6 MB 00:00 2024-01-18T07:15:11.418 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 15 MB/s | 2.5 MB 00:00 2024-01-18T07:15:11.438 INFO:teuthology.orchestra.run.smithi106.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 13 MB/s | 2.5 MB 00:00 2024-01-18T07:15:11.443 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 790 kB/s | 53 kB 00:00 2024-01-18T07:15:11.464 INFO:teuthology.orchestra.run.smithi106.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 631 kB/s | 53 kB 00:00 2024-01-18T07:15:11.468 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 821 kB/s | 41 kB 00:00 2024-01-18T07:15:11.489 INFO:teuthology.orchestra.run.smithi106.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2024-01-18T07:15:11.589 INFO:teuthology.orchestra.run.smithi106.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 22 MB/s | 8.1 MB 00:00 2024-01-18T07:15:11.589 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:15:11.589 INFO:teuthology.orchestra.run.smithi106.stdout:Total 25 MB/s | 27 MB 00:01 2024-01-18T07:15:11.593 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2024-01-18T07:15:11.594 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:15:11.594 INFO:teuthology.orchestra.run.smithi084.stdout:Total 24 MB/s | 27 MB 00:01 2024-01-18T07:15:11.855 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:15:11.856 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:15:11.876 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:15:11.876 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:15:11.877 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:15:11.877 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:15:12.131 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:15:12.134 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:15:12.135 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:15:12.138 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:15:12.816 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:15:12.860 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:15:13.013 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-18T07:15:13.077 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-18T07:15:13.200 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-18T07:15:13.280 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-18T07:15:13.515 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-18T07:15:13.721 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-18T07:15:13.766 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T07:15:13.963 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T07:15:14.317 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-18T07:15:14.509 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-18T07:15:15.322 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T07:15:15.464 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T07:15:15.555 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-18T07:15:15.703 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-18T07:15:15.799 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-18T07:15:15.923 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-18T07:15:16.068 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-18T07:15:16.163 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-18T07:15:16.484 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-18T07:15:16.544 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-18T07:15:17.406 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T07:15:17.408 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T07:15:17.622 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-18T07:15:17.640 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-18T07:15:17.816 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T07:15:17.854 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-18T07:15:18.249 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-18T07:15:18.250 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-18T07:15:18.250 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-18T07:15:18.250 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T07:15:18.250 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-18T07:15:18.309 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-18T07:15:18.310 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-18T07:15:18.771 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-18T07:15:18.771 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:18.771 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:15:18.771 INFO:teuthology.orchestra.run.smithi106.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:18.772 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-18T07:15:18.869 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:18.870 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:15:19.185 DEBUG:teuthology.parallel:result is None 2024-01-18T07:15:19.207 DEBUG:teuthology.parallel:result is None 2024-01-18T07:15:19.207 INFO:teuthology.run_tasks:Running task pexec... 2024-01-18T07:15:19.224 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-18T07:15:19.225 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-01-18T07:15:19.225 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T07:15:19.226 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi106.front.sepia.ceph.com 2024-01-18T07:15:19.226 DEBUG:teuthology.orchestra.run.smithi106:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-18T07:15:19.426 DEBUG:teuthology.parallel:result is None 2024-01-18T07:15:19.480 DEBUG:teuthology.parallel:result is None 2024-01-18T07:15:19.481 INFO:teuthology.run_tasks:Running task install... 2024-01-18T07:15:19.494 DEBUG:teuthology.task.install:project ceph 2024-01-18T07:15:19.494 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'}} 2024-01-18T07:15:19.494 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'} 2024-01-18T07:15:19.495 INFO:teuthology.task.install:Using flavor: default 2024-01-18T07:15:19.506 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-18T07:15:19.506 INFO:teuthology.task.install:extra packages: [] 2024-01-18T07:15:19.507 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-18T07:15:19.507 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:15:19.507 INFO:teuthology.packaging:ref: None 2024-01-18T07:15:19.508 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:15:19.508 INFO:teuthology.packaging:branch: None 2024-01-18T07:15:19.508 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:15:19.727 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T07:15:19.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T07:15:19.731 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-18T07:15:19.731 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:15:19.731 INFO:teuthology.packaging:ref: None 2024-01-18T07:15:19.731 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:15:19.731 INFO:teuthology.packaging:branch: None 2024-01-18T07:15:19.731 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:15:19.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T07:15:19.877 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-18T07:15:19.877 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-18T07:15:19.918 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-18T07:15:19.918 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:15:19.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-18T07:15:19.946 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-18T07:15:19.946 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:15:19.946 INFO:teuthology.packaging:ref: None 2024-01-18T07:15:19.947 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:15:19.947 INFO:teuthology.packaging:branch: None 2024-01-18T07:15:19.947 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:15:19.947 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-18T07:15:19.952 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-18T07:15:19.952 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-18T07:15:19.978 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-18T07:15:19.978 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:15:19.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-18T07:15:20.006 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-18T07:15:20.006 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:15:20.006 INFO:teuthology.packaging:ref: None 2024-01-18T07:15:20.007 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:15:20.007 INFO:teuthology.packaging:branch: None 2024-01-18T07:15:20.007 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:15:20.007 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-18T07:15:20.014 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-18T07:15:20.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-18T07:15:20.090 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-18T07:15:20.152 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-18T07:15:20.156 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-01-18T07:15:20.157 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-01-18T07:15:20.218 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-01-18T07:15:20.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-01-18T07:15:20.422 INFO:teuthology.orchestra.run.smithi106.stdout:76 files removed 2024-01-18T07:15:20.450 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw 2024-01-18T07:15:20.488 INFO:teuthology.orchestra.run.smithi084.stdout:76 files removed 2024-01-18T07:15:20.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-01-18T07:15:20.931 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 976 kB/s | 74 kB 00:00 2024-01-18T07:15:21.003 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 904 kB/s | 74 kB 00:00 2024-01-18T07:15:21.040 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 214 kB/s | 15 kB 00:00 2024-01-18T07:15:21.120 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2024-01-18T07:15:21.149 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-18T07:15:21.263 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 15 kB/s | 1.7 kB 00:00 2024-01-18T07:15:21.691 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - AppStream 51 MB/s | 26 MB 00:00 2024-01-18T07:15:21.963 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - AppStream 39 MB/s | 26 MB 00:00 2024-01-18T07:15:27.067 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-01-18T07:15:27.313 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-01-18T07:15:28.917 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras 70 kB/s | 18 kB 00:00 2024-01-18T07:15:29.339 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2024-01-18T07:15:29.418 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-18T07:15:29.807 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-18T07:15:30.103 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 8 - PowerTools 8.0 MB/s | 4.7 MB 00:00 2024-01-18T07:15:30.544 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 8 - PowerTools 7.3 MB/s | 4.7 MB 00:00 2024-01-18T07:15:31.138 INFO:teuthology.orchestra.run.smithi106.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-01-18T07:15:31.559 INFO:teuthology.orchestra.run.smithi084.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-18T07:15:32.700 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-01-18T07:15:36.451 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-01-18T07:15:37.890 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 2.4 MB/s | 16 MB 00:06 2024-01-18T07:15:39.879 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-18T07:15:39.882 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-18T07:15:39.883 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout:Install 25 Packages 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-01-18T07:15:39.884 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:15:39.885 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 59 M 2024-01-18T07:15:39.885 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:15:40.160 INFO:teuthology.orchestra.run.smithi084.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2024-01-18T07:15:40.185 INFO:teuthology.orchestra.run.smithi084.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 997 kB/s | 25 kB 00:00 2024-01-18T07:15:40.227 INFO:teuthology.orchestra.run.smithi084.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-18T07:15:40.261 INFO:teuthology.orchestra.run.smithi084.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-18T07:15:40.378 INFO:teuthology.orchestra.run.smithi084.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2024-01-18T07:15:40.395 INFO:teuthology.orchestra.run.smithi084.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-18T07:15:40.420 INFO:teuthology.orchestra.run.smithi084.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-18T07:15:40.454 INFO:teuthology.orchestra.run.smithi084.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-01-18T07:15:40.488 INFO:teuthology.orchestra.run.smithi084.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-18T07:15:40.513 INFO:teuthology.orchestra.run.smithi084.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-01-18T07:15:40.538 INFO:teuthology.orchestra.run.smithi084.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-18T07:15:40.714 INFO:teuthology.orchestra.run.smithi084.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-01-18T07:15:40.739 INFO:teuthology.orchestra.run.smithi084.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 725 kB/s | 145 kB 00:00 2024-01-18T07:15:40.789 INFO:teuthology.orchestra.run.smithi084.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2024-01-18T07:15:40.823 INFO:teuthology.orchestra.run.smithi084.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2024-01-18T07:15:40.856 INFO:teuthology.orchestra.run.smithi084.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 4.6 MB/s | 677 kB 00:00 2024-01-18T07:15:41.107 INFO:teuthology.orchestra.run.smithi084.stdout:(17/27): ceph-common-16.2.4-0.el8.x86_64.rpm 22 MB/s | 24 MB 00:01 2024-01-18T07:15:41.132 INFO:teuthology.orchestra.run.smithi084.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 136 kB/s | 42 kB 00:00 2024-01-18T07:15:41.157 INFO:teuthology.orchestra.run.smithi084.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 156 kB/s | 47 kB 00:00 2024-01-18T07:15:41.182 INFO:teuthology.orchestra.run.smithi084.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.0 MB/s | 80 kB 00:00 2024-01-18T07:15:41.613 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 420 kB/s | 24 kB 00:00 2024-01-18T07:15:41.741 INFO:teuthology.orchestra.run.smithi084.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 191 kB/s | 116 kB 00:00 2024-01-18T07:15:41.766 INFO:teuthology.orchestra.run.smithi084.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 310 kB/s | 181 kB 00:00 2024-01-18T07:15:41.858 INFO:teuthology.orchestra.run.smithi084.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 437 kB/s | 306 kB 00:00 2024-01-18T07:15:41.883 INFO:teuthology.orchestra.run.smithi084.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 414 kB/s | 59 kB 00:00 2024-01-18T07:15:41.909 INFO:teuthology.orchestra.run.smithi084.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 527 kB/s | 75 kB 00:00 2024-01-18T07:15:42.015 INFO:teuthology.orchestra.run.smithi084.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2024-01-18T07:15:42.084 INFO:teuthology.orchestra.run.smithi084.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 20 MB/s | 4.0 MB 00:00 2024-01-18T07:15:42.086 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:15:42.087 INFO:teuthology.orchestra.run.smithi084.stdout:Total 27 MB/s | 59 MB 00:02 2024-01-18T07:15:42.155 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:15:42.203 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:15:42.203 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:15:42.547 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:15:42.550 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:15:43.727 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:15:43.952 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-18T07:15:44.390 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-18T07:15:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T07:15:44.995 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T07:15:45.224 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:15:45.318 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:15:45.509 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-18T07:15:45.512 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-18T07:15:45.513 INFO:teuthology.orchestra.run.smithi106.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout:Install 25 Packages 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 2 Packages 2024-01-18T07:15:45.514 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:15:45.515 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 59 M 2024-01-18T07:15:45.515 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:15:45.578 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T07:15:45.724 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:15:45.814 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:15:46.025 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-18T07:15:46.081 INFO:teuthology.orchestra.run.smithi106.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 42 MB/s | 6.0 MB 00:00 2024-01-18T07:15:46.106 INFO:teuthology.orchestra.run.smithi106.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2024-01-18T07:15:46.148 INFO:teuthology.orchestra.run.smithi106.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-18T07:15:46.182 INFO:teuthology.orchestra.run.smithi106.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-18T07:15:46.274 INFO:teuthology.orchestra.run.smithi106.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-01-18T07:15:46.322 INFO:teuthology.orchestra.run.smithi106.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 921 kB/s | 44 kB 00:00 2024-01-18T07:15:46.341 INFO:teuthology.orchestra.run.smithi106.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.3 MB/s | 80 kB 00:00 2024-01-18T07:15:46.366 INFO:teuthology.orchestra.run.smithi106.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-01-18T07:15:46.509 INFO:teuthology.orchestra.run.smithi106.stdout:(9/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2024-01-18T07:15:46.534 INFO:teuthology.orchestra.run.smithi106.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 387 kB 00:00 2024-01-18T07:15:46.559 INFO:teuthology.orchestra.run.smithi106.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-18T07:15:46.584 INFO:teuthology.orchestra.run.smithi106.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.2 MB/s | 368 kB 00:00 2024-01-18T07:15:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-18T07:15:46.643 INFO:teuthology.orchestra.run.smithi106.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 145 kB 00:00 2024-01-18T07:15:46.685 INFO:teuthology.orchestra.run.smithi106.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-01-18T07:15:46.718 INFO:teuthology.orchestra.run.smithi106.stdout:(15/27): librdkafka-1.6.1-1.el8.x86_64.rpm 4.9 MB/s | 677 kB 00:00 2024-01-18T07:15:46.744 INFO:teuthology.orchestra.run.smithi106.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 748 kB/s | 44 kB 00:00 2024-01-18T07:15:46.782 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:15:46.845 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:15:46.969 INFO:teuthology.orchestra.run.smithi106.stdout:(17/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2024-01-18T07:15:47.020 INFO:teuthology.orchestra.run.smithi106.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 171 kB/s | 47 kB 00:00 2024-01-18T07:15:47.037 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-18T07:15:47.045 INFO:teuthology.orchestra.run.smithi106.stdout:(19/27): daxctl-libs-71.1-7.el8.x86_64.rpm 129 kB/s | 42 kB 00:00 2024-01-18T07:15:47.070 INFO:teuthology.orchestra.run.smithi106.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 799 kB/s | 80 kB 00:00 2024-01-18T07:15:47.240 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T07:15:47.301 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T07:15:47.429 INFO:teuthology.orchestra.run.smithi106.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 284 kB/s | 116 kB 00:00 2024-01-18T07:15:47.470 INFO:teuthology.orchestra.run.smithi106.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 452 kB/s | 181 kB 00:00 2024-01-18T07:15:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T07:15:47.504 INFO:teuthology.orchestra.run.smithi106.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 667 kB/s | 306 kB 00:00 2024-01-18T07:15:47.529 INFO:teuthology.orchestra.run.smithi106.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 585 kB/s | 59 kB 00:00 2024-01-18T07:15:47.580 INFO:teuthology.orchestra.run.smithi106.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 689 kB/s | 75 kB 00:00 2024-01-18T07:15:47.655 INFO:teuthology.orchestra.run.smithi106.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-01-18T07:15:47.699 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T07:15:47.747 INFO:teuthology.orchestra.run.smithi106.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2024-01-18T07:15:47.749 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:15:47.749 INFO:teuthology.orchestra.run.smithi106.stdout:Total 26 MB/s | 59 MB 00:02 2024-01-18T07:15:47.819 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:15:47.867 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:15:47.867 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:15:47.881 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T07:15:47.973 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T07:15:48.164 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:15:48.206 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:15:48.209 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:15:48.602 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:15:48.773 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T07:15:48.867 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T07:15:49.060 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-18T07:15:49.293 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:15:49.308 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-18T07:15:49.448 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T07:15:49.502 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-18T07:15:49.554 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T07:15:49.714 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-18T07:15:49.863 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T07:15:50.287 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-18T07:15:50.318 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-18T07:15:50.648 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-18T07:15:50.704 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:15:50.832 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:15:50.833 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-18T07:15:51.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-18T07:15:51.086 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T07:15:51.246 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:15:51.311 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:15:51.547 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-18T07:15:52.113 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-18T07:15:52.254 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:15:52.325 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:15:52.492 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-18T07:15:52.679 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T07:15:52.751 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-18T07:15:52.918 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T07:15:53.104 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T07:15:53.271 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T07:15:53.346 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-18T07:15:53.546 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:15:53.680 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:53.984 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:15:54.129 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T07:15:54.212 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-18T07:15:54.416 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-18T07:15:54.630 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-18T07:15:54.655 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:54.762 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T07:15:54.830 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-18T07:15:54.950 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T07:15:55.004 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-18T07:15:55.055 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T07:15:55.170 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-18T07:15:55.221 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:15:55.369 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-18T07:15:55.571 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-18T07:15:58.373 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:58.587 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:59.301 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-18T07:15:59.594 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T07:15:59.696 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-18T07:15:59.870 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /run 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-01-18T07:16:14.261 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-01-18T07:16:14.262 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-01-18T07:16:14.262 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-01-18T07:16:14.262 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /dev 2024-01-18T07:16:14.262 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:15.587 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T07:16:15.665 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T07:16:15.666 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T07:16:15.911 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T07:16:15.911 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:16.612 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:16.612 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-18T07:16:16.612 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:16:16.613 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-18T07:16:16.614 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-18T07:16:17.717 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:17.717 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: leveldb-1.22-1.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.718 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.719 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:17.720 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:17.720 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:18.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-01-18T07:16:18.753 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:42 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /run 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-01-18T07:16:19.044 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /dev 2024-01-18T07:16:19.045 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:20.236 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-18T07:16:20.237 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-18T07:16:20.238 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:20.239 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:16:20.239 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:20.239 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-01-18T07:16:20.239 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:20.240 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 53 M 2024-01-18T07:16:20.240 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 229 M 2024-01-18T07:16:20.240 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:16:20.357 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 2.6 MB/s | 203 kB 00:00 2024-01-18T07:16:20.392 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T07:16:20.403 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.3 MB/s | 169 kB 00:00 2024-01-18T07:16:20.432 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-01-18T07:16:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-18T07:16:20.450 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T07:16:20.466 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 71 kB 00:00 2024-01-18T07:16:20.508 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.0 MB/s | 323 kB 00:00 2024-01-18T07:16:20.694 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-18T07:16:20.695 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:21.083 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-01-18T07:16:21.085 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:21.085 INFO:teuthology.orchestra.run.smithi084.stdout:Total 62 MB/s | 53 MB 00:00 2024-01-18T07:16:21.127 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:16:21.148 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:16:21.148 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:16:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:21.400 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-18T07:16:21.401 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-18T07:16:21.402 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-18T07:16:21.460 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:16:21.464 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:16:21.961 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:16:22.205 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-18T07:16:22.286 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-18T07:16:22.286 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:22.286 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: leveldb-1.22-1.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.287 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:22.288 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:16:22.441 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-18T07:16:22.592 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-test 2024-01-18T07:16:22.616 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T07:16:22.695 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T07:16:22.864 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-18T07:16:23.032 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T07:16:23.315 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:42 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:16:24.802 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:16:24.803 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:24.803 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:16:24.803 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:24.803 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:16:24.803 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout:Install 6 Packages 2024-01-18T07:16:24.804 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:24.805 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 53 M 2024-01-18T07:16:24.805 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 229 M 2024-01-18T07:16:24.805 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:16:25.058 INFO:teuthology.orchestra.run.smithi106.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-01-18T07:16:25.083 INFO:teuthology.orchestra.run.smithi106.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.1 MB/s | 203 kB 00:00 2024-01-18T07:16:25.134 INFO:teuthology.orchestra.run.smithi106.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-01-18T07:16:25.159 INFO:teuthology.orchestra.run.smithi106.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-01-18T07:16:25.184 INFO:teuthology.orchestra.run.smithi106.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-01-18T07:16:25.784 INFO:teuthology.orchestra.run.smithi106.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-01-18T07:16:25.786 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:25.787 INFO:teuthology.orchestra.run.smithi106.stdout:Total 54 MB/s | 53 MB 00:00 2024-01-18T07:16:25.830 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:16:25.851 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:16:25.851 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:16:26.160 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:16:26.163 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:16:26.648 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:16:26.877 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-18T07:16:27.078 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-18T07:16:27.225 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T07:16:27.305 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-18T07:16:27.482 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-18T07:16:27.633 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T07:16:27.929 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T07:16:28.196 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-18T07:16:28.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-18T07:16:28.902 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:28.903 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:29.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-01-18T07:16:29.850 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:53 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:31.357 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-18T07:16:31.361 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-18T07:16:31.362 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-18T07:16:31.363 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout:Install 32 Packages 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 M 2024-01-18T07:16:31.364 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 111 M 2024-01-18T07:16:31.365 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:16:31.717 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 111 kB/s | 6.3 kB 00:00 2024-01-18T07:16:31.767 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-18T07:16:31.817 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-18T07:16:31.851 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-18T07:16:31.943 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-01-18T07:16:31.985 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 939 kB/s | 39 kB 00:00 2024-01-18T07:16:32.010 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2024-01-18T07:16:32.044 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2024-01-18T07:16:32.069 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 972 kB/s | 57 kB 00:00 2024-01-18T07:16:32.303 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 34 MB/s | 18 MB 00:00 2024-01-18T07:16:32.328 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.6 MB/s | 457 kB 00:00 2024-01-18T07:16:32.353 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 876 kB/s | 249 kB 00:00 2024-01-18T07:16:32.412 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-01-18T07:16:32.437 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 4.7 MB/s | 641 kB 00:00 2024-01-18T07:16:32.502 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-18T07:16:32.563 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 208 kB/s | 44 kB 00:00 2024-01-18T07:16:32.638 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 820 kB/s | 185 kB 00:00 2024-01-18T07:16:32.680 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 712 kB/s | 173 kB 00:00 2024-01-18T07:16:32.705 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 166 kB/s | 11 kB 00:00 2024-01-18T07:16:32.714 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T07:16:32.738 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 300 kB/s | 18 kB 00:00 2024-01-18T07:16:32.772 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 728 kB/s | 49 kB 00:00 2024-01-18T07:16:32.797 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-01-18T07:16:32.844 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 557 kB/s | 59 kB 00:00 2024-01-18T07:16:32.881 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-01-18T07:16:32.948 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 180 kB/s | 19 kB 00:00 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-18T07:16:33.005 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-18T07:16:33.099 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 110 kB/s | 24 kB 00:00 2024-01-18T07:16:33.124 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): python3-portend-2.6-1.el8.noarch.rpm 48 kB/s | 16 kB 00:00 2024-01-18T07:16:33.174 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 130 kB/s | 29 kB 00:00 2024-01-18T07:16:33.233 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 206 kB/s | 27 kB 00:00 2024-01-18T07:16:33.258 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.9 MB/s | 255 kB 00:00 2024-01-18T07:16:33.283 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2024-01-18T07:16:33.350 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-01-18T07:16:33.375 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 600 kB/s | 85 kB 00:00 2024-01-18T07:16:33.376 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:33.377 INFO:teuthology.orchestra.run.smithi084.stdout:Total 15 MB/s | 31 MB 00:02 2024-01-18T07:16:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-18T07:16:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:33.448 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:16:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-18T07:16:33.448 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-18T07:16:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-18T07:16:33.449 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:33.449 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:16:33.459 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:16:33.497 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:16:33.498 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:16:33.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph 2024-01-18T07:16:33.803 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:16:33.806 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:16:34.392 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:00:53 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:16:34.959 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:16:35.176 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-18T07:16:35.412 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-18T07:16:35.638 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-18T07:16:35.819 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-18T07:16:35.881 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-18T07:16:35.884 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-18T07:16:35.885 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-18T07:16:35.886 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout:Install 32 Packages 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:35.887 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 31 M 2024-01-18T07:16:35.888 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 111 M 2024-01-18T07:16:35.888 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:16:36.017 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-18T07:16:36.211 INFO:teuthology.orchestra.run.smithi106.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 113 kB/s | 6.3 kB 00:00 2024-01-18T07:16:36.215 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-18T07:16:36.261 INFO:teuthology.orchestra.run.smithi106.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-18T07:16:36.312 INFO:teuthology.orchestra.run.smithi106.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-01-18T07:16:36.337 INFO:teuthology.orchestra.run.smithi106.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-18T07:16:36.424 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-18T07:16:36.429 INFO:teuthology.orchestra.run.smithi106.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-01-18T07:16:36.454 INFO:teuthology.orchestra.run.smithi106.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-01-18T07:16:36.487 INFO:teuthology.orchestra.run.smithi106.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2024-01-18T07:16:36.513 INFO:teuthology.orchestra.run.smithi106.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 463 kB/s | 39 kB 00:00 2024-01-18T07:16:36.538 INFO:teuthology.orchestra.run.smithi106.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-01-18T07:16:36.588 INFO:teuthology.orchestra.run.smithi106.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.0 MB/s | 457 kB 00:00 2024-01-18T07:16:36.622 INFO:teuthology.orchestra.run.smithi106.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.9 MB/s | 249 kB 00:00 2024-01-18T07:16:36.635 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-18T07:16:36.814 INFO:teuthology.orchestra.run.smithi106.stdout:(12/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 33 MB/s | 18 MB 00:00 2024-01-18T07:16:36.839 INFO:teuthology.orchestra.run.smithi106.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 503 kB/s | 109 kB 00:00 2024-01-18T07:16:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-18T07:16:36.881 INFO:teuthology.orchestra.run.smithi106.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 2.1 MB/s | 641 kB 00:00 2024-01-18T07:16:37.073 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-18T07:16:37.157 INFO:teuthology.orchestra.run.smithi106.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 127 kB/s | 44 kB 00:00 2024-01-18T07:16:37.257 INFO:teuthology.orchestra.run.smithi106.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 444 kB/s | 185 kB 00:00 2024-01-18T07:16:37.282 INFO:teuthology.orchestra.run.smithi106.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 430 kB/s | 173 kB 00:00 2024-01-18T07:16:37.318 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-18T07:16:37.358 INFO:teuthology.orchestra.run.smithi106.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-01-18T07:16:37.383 INFO:teuthology.orchestra.run.smithi106.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-01-18T07:16:37.425 INFO:teuthology.orchestra.run.smithi106.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-01-18T07:16:37.458 INFO:teuthology.orchestra.run.smithi106.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2024-01-18T07:16:37.484 INFO:teuthology.orchestra.run.smithi106.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 584 kB/s | 59 kB 00:00 2024-01-18T07:16:37.526 INFO:teuthology.orchestra.run.smithi106.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-01-18T07:16:37.577 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-18T07:16:37.588 INFO:teuthology.orchestra.run.smithi106.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 120 kB/s | 16 kB 00:00 2024-01-18T07:16:37.618 INFO:teuthology.orchestra.run.smithi106.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2024-01-18T07:16:37.643 INFO:teuthology.orchestra.run.smithi106.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2024-01-18T07:16:37.685 INFO:teuthology.orchestra.run.smithi106.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 302 kB/s | 29 kB 00:00 2024-01-18T07:16:37.719 INFO:teuthology.orchestra.run.smithi106.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-01-18T07:16:37.744 INFO:teuthology.orchestra.run.smithi106.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-01-18T07:16:37.819 INFO:teuthology.orchestra.run.smithi106.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 852 kB/s | 85 kB 00:00 2024-01-18T07:16:37.845 INFO:teuthology.orchestra.run.smithi106.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-01-18T07:16:37.870 INFO:teuthology.orchestra.run.smithi106.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-01-18T07:16:37.871 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:37.871 INFO:teuthology.orchestra.run.smithi106.stdout:Total 16 MB/s | 31 MB 00:01 2024-01-18T07:16:37.937 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-18T07:16:37.951 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:16:37.990 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:16:37.990 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:16:38.234 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-18T07:16:38.298 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:16:38.301 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:16:38.403 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-18T07:16:38.696 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-18T07:16:38.905 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T07:16:39.234 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-18T07:16:39.410 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:16:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-18T07:16:39.625 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-18T07:16:39.657 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-18T07:16:39.831 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-18T07:16:39.904 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-18T07:16:40.070 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-18T07:16:40.190 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-18T07:16:40.296 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-18T07:16:40.440 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-18T07:16:40.469 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-18T07:16:40.651 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-18T07:16:40.677 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-18T07:16:40.817 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-18T07:16:40.979 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-18T07:16:41.024 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-18T07:16:41.213 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-18T07:16:41.360 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-18T07:16:41.425 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-18T07:16:41.629 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-18T07:16:41.769 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-18T07:16:41.863 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-18T07:16:42.006 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T07:16:42.266 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-18T07:16:42.527 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-18T07:16:42.689 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-18T07:16:42.942 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-18T07:16:43.132 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T07:16:43.429 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-18T07:16:43.542 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T07:16:43.646 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-18T07:16:43.764 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T07:16:43.809 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-18T07:16:44.040 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-18T07:16:44.197 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T07:16:44.310 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-18T07:16:44.431 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T07:16:44.541 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-18T07:16:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T07:16:44.771 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-18T07:16:44.861 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T07:16:44.878 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T07:16:45.035 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T07:16:45.125 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-18T07:16:45.420 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-18T07:16:45.432 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T07:16:45.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-18T07:16:45.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-18T07:16:45.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-18T07:16:45.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-18T07:16:45.834 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-18T07:16:46.038 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T07:16:46.229 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-18T07:16:46.230 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-2.6-1.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-18T07:16:46.231 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:46.232 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:46.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-01-18T07:16:47.281 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:11 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:47.571 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-18T07:16:47.763 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T07:16:48.196 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-18T07:16:48.429 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T07:16:48.651 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-18T07:16:48.724 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:16:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:16:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:48.805 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T07:16:48.822 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-18T07:16:48.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-01-18T07:16:48.946 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T07:16:49.377 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-18T07:16:49.377 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-18T07:16:49.377 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-18T07:16:49.377 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-18T07:16:49.377 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-18T07:16:49.378 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-18T07:16:49.379 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-18T07:16:49.629 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:13 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:50.298 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-18T07:16:50.298 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-18T07:16:50.299 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-18T07:16:50.300 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-2.6-1.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-18T07:16:50.301 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:50.302 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:16:50.511 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-base 2024-01-18T07:16:51.112 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 71 k 2024-01-18T07:16:51.113 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 290 k 2024-01-18T07:16:51.114 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:16:51.153 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-01-18T07:16:51.154 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:51.154 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-01-18T07:16:51.154 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:16:51.161 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:16:51.161 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:16:51.163 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:16:51.166 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:16:51.231 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:16:51.242 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:10 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:16:51.695 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:51.900 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:52.133 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:52.571 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:52.571 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:52.571 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:16:52.571 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-18T07:16:52.571 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:52.572 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:52.655 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:16:52.701 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:16:52.702 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:16:52.702 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:16:52.795 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install cephadm 2024-01-18T07:16:52.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-01-18T07:16:53.515 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:12 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:16:53.594 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:17 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:55.015 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:16:55.016 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:55.016 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:16:55.016 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:55.016 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 71 k 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 290 k 2024-01-18T07:16:55.017 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:16:55.057 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:16:55.058 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:16:55.059 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:16:55.059 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:55.059 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 163 k 2024-01-18T07:16:55.059 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 587 k 2024-01-18T07:16:55.059 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:16:55.065 INFO:teuthology.orchestra.run.smithi106.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-01-18T07:16:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:55.066 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.4 MB/s | 71 kB 00:00 2024-01-18T07:16:55.067 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:16:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:16:55.073 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:16:55.075 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:16:55.078 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:16:55.125 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:16:55.566 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 321 kB/s | 163 kB 00:00 2024-01-18T07:16:55.567 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:55.567 INFO:teuthology.orchestra.run.smithi084.stdout:Total 320 kB/s | 163 kB 00:00 2024-01-18T07:16:55.568 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:16:55.580 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:16:55.580 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:16:55.619 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:55.634 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:16:55.637 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:16:55.791 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:55.896 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:16:56.025 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:56.075 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:16:56.403 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-18T07:16:56.404 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:56.404 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:16:56.404 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-18T07:16:56.404 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:56.404 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:16:56.496 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:16:56.630 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-immutable-object-cache 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:16:56.921 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:57.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-01-18T07:16:57.360 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:16 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:16:57.885 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:21 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:16:58.812 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repo Size 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:58.813 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 163 k 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 587 k 2024-01-18T07:16:58.814 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:16:58.853 INFO:teuthology.orchestra.run.smithi106.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.0 MB/s | 163 kB 00:00 2024-01-18T07:16:58.854 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:16:58.854 INFO:teuthology.orchestra.run.smithi106.stdout:Total 3.9 MB/s | 163 kB 00:00 2024-01-18T07:16:58.855 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:16:58.865 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:16:58.865 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:16:58.911 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:16:58.914 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:16:59.022 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:16:59.212 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:16:59.294 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:16:59.340 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:16:59.341 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:16:59.341 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:16:59.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-01-18T07:16:59.636 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:00.027 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:00.187 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:24 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:00.248 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr 2024-01-18T07:17:00.966 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:19 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:01.660 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:01.660 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:01.661 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-01-18T07:17:01.662 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:01.662 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.0 M 2024-01-18T07:17:01.662 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 15 M 2024-01-18T07:17:01.662 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:01.969 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 381 kB/s | 24 kB 00:00 2024-01-18T07:17:01.994 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 101 kB/s | 8.9 kB 00:00 2024-01-18T07:17:02.070 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-01-18T07:17:02.128 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 268 kB/s | 43 kB 00:00 2024-01-18T07:17:02.237 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-01-18T07:17:02.405 INFO:teuthology.orchestra.run.smithi106.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:02.451 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:02.452 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:02.452 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:02.471 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 491 kB/s | 196 kB 00:00 2024-01-18T07:17:02.472 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:02.472 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2024-01-18T07:17:02.482 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:02.492 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:02.492 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:02.615 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-dashboard 2024-01-18T07:17:02.655 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:02.658 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:02.956 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:03.207 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-18T07:17:03.341 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:22 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:03.484 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-18T07:17:03.708 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-18T07:17:03.838 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-18T07:17:04.810 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:04.811 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:04.811 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:17:04.811 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout:Install 6 Packages 2024-01-18T07:17:04.812 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:04.813 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 3.0 M 2024-01-18T07:17:04.813 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 15 M 2024-01-18T07:17:04.813 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:04.937 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-18T07:17:05.049 INFO:teuthology.orchestra.run.smithi106.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 429 kB/s | 24 kB 00:00 2024-01-18T07:17:05.075 INFO:teuthology.orchestra.run.smithi106.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 110 kB/s | 8.9 kB 00:00 2024-01-18T07:17:05.133 INFO:teuthology.orchestra.run.smithi106.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-01-18T07:17:05.184 INFO:teuthology.orchestra.run.smithi106.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 318 kB/s | 43 kB 00:00 2024-01-18T07:17:05.225 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:05.351 INFO:teuthology.orchestra.run.smithi106.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 122 kB/s | 34 kB 00:00 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-18T07:17:05.515 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-18T07:17:05.551 INFO:teuthology.orchestra.run.smithi106.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 471 kB/s | 196 kB 00:00 2024-01-18T07:17:05.553 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:05.553 INFO:teuthology.orchestra.run.smithi106.stdout:Total 4.0 MB/s | 3.0 MB 00:00 2024-01-18T07:17:05.563 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:05.572 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:05.572 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:05.721 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:05.724 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:05.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-18T07:17:05.928 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:05.929 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:05.992 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:06.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-18T07:17:06.200 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-18T07:17:06.462 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-18T07:17:06.655 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-18T07:17:06.776 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-18T07:17:06.788 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:30 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:07.735 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-18T07:17:07.993 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:08.273 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-18T07:17:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-18T07:17:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-18T07:17:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-18T07:17:08.274 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-18T07:17:08.277 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:08.278 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-01-18T07:17:08.278 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:17:08.278 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-01-18T07:17:08.278 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:08.278 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-18T07:17:08.279 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-18T07:17:08.280 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:08.280 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:08.280 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-01-18T07:17:08.280 INFO:teuthology.orchestra.run.smithi084.stdout:Install 8 Packages 2024-01-18T07:17:08.280 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:08.281 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2024-01-18T07:17:08.281 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 210 M 2024-01-18T07:17:08.281 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:08.525 INFO:teuthology.orchestra.run.smithi084.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 52 MB/s | 8.6 MB 00:00 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-18T07:17:08.651 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-18T07:17:08.652 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-18T07:17:08.652 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:08.652 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:08.666 INFO:teuthology.orchestra.run.smithi084.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2024-01-18T07:17:08.700 INFO:teuthology.orchestra.run.smithi084.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2024-01-18T07:17:08.767 INFO:teuthology.orchestra.run.smithi084.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-01-18T07:17:08.769 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-18T07:17:08.834 INFO:teuthology.orchestra.run.smithi084.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2024-01-18T07:17:09.026 INFO:teuthology.orchestra.run.smithi084.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 44 MB/s | 14 MB 00:00 2024-01-18T07:17:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 795 kB/s | 172 kB 00:00 2024-01-18T07:17:09.085 INFO:teuthology.orchestra.run.smithi084.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2024-01-18T07:17:09.086 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:09.086 INFO:teuthology.orchestra.run.smithi084.stdout:Total 46 MB/s | 37 MB 00:00 2024-01-18T07:17:09.315 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:09.334 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:09.334 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:09.491 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:28 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:09.654 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:09.657 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:10.314 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:10.480 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T07:17:10.625 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T07:17:10.788 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T07:17:10.991 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-18T07:17:10.993 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout:Install 8 Packages 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:10.994 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 37 M 2024-01-18T07:17:10.995 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 210 M 2024-01-18T07:17:10.995 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T07:17:11.530 INFO:teuthology.orchestra.run.smithi106.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2024-01-18T07:17:11.672 INFO:teuthology.orchestra.run.smithi106.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-01-18T07:17:11.705 INFO:teuthology.orchestra.run.smithi106.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2024-01-18T07:17:11.774 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:11.906 INFO:teuthology.orchestra.run.smithi106.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 71 MB/s | 14 MB 00:00 2024-01-18T07:17:11.973 INFO:teuthology.orchestra.run.smithi106.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.2 MB/s | 4.9 MB 00:00 2024-01-18T07:17:12.031 INFO:teuthology.orchestra.run.smithi106.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.4 MB/s | 3.7 MB 00:00 2024-01-18T07:17:12.132 INFO:teuthology.orchestra.run.smithi106.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-01-18T07:17:12.165 INFO:teuthology.orchestra.run.smithi106.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.4 MB/s | 645 kB 00:00 2024-01-18T07:17:12.166 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:12.166 INFO:teuthology.orchestra.run.smithi106.stdout:Total 32 MB/s | 37 MB 00:01 2024-01-18T07:17:12.396 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:12.415 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:12.415 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:12.570 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:12.743 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:12.746 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:12.779 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T07:17:13.341 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:13.463 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T07:17:13.502 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T07:17:13.651 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-18T07:17:13.793 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T07:17:13.799 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T07:17:14.054 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T07:17:14.387 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-18T07:17:14.568 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:15.181 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:15.368 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T07:17:15.814 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T07:17:16.059 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-18T07:17:16.404 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T07:17:16.635 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T07:17:17.128 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-18T07:17:18.410 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-18T07:17:18.448 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-18T07:17:18.776 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-18T07:17:19.209 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:19.210 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:19.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-01-18T07:17:19.780 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-18T07:17:19.963 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T07:17:20.048 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:44 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-18T07:17:20.297 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-18T07:17:20.702 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:20.703 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:20.803 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-rook 2024-01-18T07:17:21.525 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:40 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:21.539 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-18T07:17:21.541 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-01-18T07:17:21.542 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:21.543 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 1.9 M 2024-01-18T07:17:21.543 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2024-01-18T07:17:21.543 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:21.842 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 612 kB/s | 27 kB 00:00 2024-01-18T07:17:21.867 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 386 kB/s | 27 kB 00:00 2024-01-18T07:17:21.892 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 212 kB/s | 20 kB 00:00 2024-01-18T07:17:21.967 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-01-18T07:17:21.993 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 839 kB/s | 126 kB 00:00 2024-01-18T07:17:22.018 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.2 MB/s | 155 kB 00:00 2024-01-18T07:17:22.043 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 563 kB/s | 43 kB 00:00 2024-01-18T07:17:22.168 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 108 kB/s | 16 kB 00:00 2024-01-18T07:17:22.194 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 164 kB/s | 33 kB 00:00 2024-01-18T07:17:22.257 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 379 kB/s | 81 kB 00:00 2024-01-18T07:17:22.303 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-01-18T07:17:22.327 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 877 kB/s | 61 kB 00:00 2024-01-18T07:17:22.461 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.0 MB/s | 1.2 MB 00:00 2024-01-18T07:17:22.462 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:22.462 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2024-01-18T07:17:22.515 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:22.530 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:22.530 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:22.616 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:22.619 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:22.988 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-18T07:17:22.990 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout:Install 13 Packages 2024-01-18T07:17:22.991 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:22.992 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 1.9 M 2024-01-18T07:17:22.992 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 24 M 2024-01-18T07:17:22.992 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:23.136 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:23.182 INFO:teuthology.orchestra.run.smithi106.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 588 kB/s | 27 kB 00:00 2024-01-18T07:17:23.207 INFO:teuthology.orchestra.run.smithi106.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 375 kB/s | 27 kB 00:00 2024-01-18T07:17:23.232 INFO:teuthology.orchestra.run.smithi106.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 208 kB/s | 20 kB 00:00 2024-01-18T07:17:23.299 INFO:teuthology.orchestra.run.smithi106.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.6 MB/s | 110 kB 00:00 2024-01-18T07:17:23.404 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-18T07:17:23.413 INFO:teuthology.orchestra.run.smithi106.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 546 kB/s | 126 kB 00:00 2024-01-18T07:17:23.551 INFO:teuthology.orchestra.run.smithi106.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 488 kB/s | 155 kB 00:00 2024-01-18T07:17:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-18T07:17:23.654 INFO:teuthology.orchestra.run.smithi106.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 120 kB/s | 43 kB 00:00 2024-01-18T07:17:23.859 INFO:teuthology.orchestra.run.smithi106.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 53 kB/s | 16 kB 00:00 2024-01-18T07:17:23.876 INFO:teuthology.orchestra.run.smithi106.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 71 kB/s | 33 kB 00:00 2024-01-18T07:17:23.968 INFO:teuthology.orchestra.run.smithi106.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 259 kB/s | 81 kB 00:00 2024-01-18T07:17:23.995 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-18T07:17:24.052 INFO:teuthology.orchestra.run.smithi106.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-01-18T07:17:24.077 INFO:teuthology.orchestra.run.smithi106.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 564 kB/s | 61 kB 00:00 2024-01-18T07:17:24.250 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-18T07:17:24.343 INFO:teuthology.orchestra.run.smithi106.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-01-18T07:17:24.344 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:24.344 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2024-01-18T07:17:24.400 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:24.416 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:24.416 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:24.497 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-18T07:17:24.502 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:24.505 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:24.730 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-18T07:17:24.980 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:25.007 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-18T07:17:25.234 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-18T07:17:25.272 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-18T07:17:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-18T07:17:25.603 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-18T07:17:25.823 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-18T07:17:26.070 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-18T07:17:26.322 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-18T07:17:26.345 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-18T07:17:26.588 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-18T07:17:26.650 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-18T07:17:26.830 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-18T07:17:26.833 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-18T07:17:27.050 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-18T07:17:27.251 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-18T07:17:27.546 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-18T07:17:27.547 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-18T07:17:27.547 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-18T07:17:27.881 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-18T07:17:27.998 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-18T07:17:27.998 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:27.998 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-18T07:17:27.999 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-18T07:17:28.000 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-18T07:17:28.000 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:28.000 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:28.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-01-18T07:17:28.165 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-18T07:17:28.365 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-18T07:17:28.541 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-18T07:17:28.716 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T07:17:28.853 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:52 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-18T07:17:29.034 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-18T07:17:29.035 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-18T07:17:29.485 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:29.486 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:29.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-mgr-cephadm 2024-01-18T07:17:30.342 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:30.343 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:49 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:30.343 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:30.344 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:30.345 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-01-18T07:17:30.345 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:30.345 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 5.8 M 2024-01-18T07:17:30.345 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 24 M 2024-01-18T07:17:30.345 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:30.778 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2024-01-18T07:17:30.837 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.8 MB/s | 538 kB 00:00 2024-01-18T07:17:30.929 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2024-01-18T07:17:30.954 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 121 kB/s | 21 kB 00:00 2024-01-18T07:17:31.146 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 225 kB/s | 49 kB 00:00 2024-01-18T07:17:31.205 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 967 kB/s | 355 kB 00:00 2024-01-18T07:17:31.205 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:31.205 INFO:teuthology.orchestra.run.smithi084.stdout:Total 6.8 MB/s | 5.8 MB 00:00 2024-01-18T07:17:31.257 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:31.267 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:31.267 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:31.380 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:31.383 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:31.683 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:31.816 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-18T07:17:31.817 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout:Install 6 Packages 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:31.818 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 5.8 M 2024-01-18T07:17:31.819 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 24 M 2024-01-18T07:17:31.819 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:31.974 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-18T07:17:32.150 INFO:teuthology.orchestra.run.smithi106.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-01-18T07:17:32.241 INFO:teuthology.orchestra.run.smithi106.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.8 MB/s | 538 kB 00:00 2024-01-18T07:17:32.286 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-18T07:17:32.326 INFO:teuthology.orchestra.run.smithi106.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2024-01-18T07:17:32.443 INFO:teuthology.orchestra.run.smithi106.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 72 kB/s | 21 kB 00:00 2024-01-18T07:17:32.635 INFO:teuthology.orchestra.run.smithi106.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 158 kB/s | 49 kB 00:00 2024-01-18T07:17:32.760 INFO:teuthology.orchestra.run.smithi106.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 685 kB/s | 355 kB 00:00 2024-01-18T07:17:32.760 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:32.761 INFO:teuthology.orchestra.run.smithi106.stdout:Total 6.2 MB/s | 5.8 MB 00:00 2024-01-18T07:17:32.813 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:32.822 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:32.822 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:32.923 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:32.927 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:33.206 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:33.249 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-18T07:17:33.473 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-18T07:17:33.620 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-18T07:17:33.756 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-18T07:17:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-18T07:17:34.141 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-18T07:17:34.426 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-18T07:17:34.699 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-18T07:17:34.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-18T07:17:34.813 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:34.814 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:34.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-01-18T07:17:35.073 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-18T07:17:35.298 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-18T07:17:35.474 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:35.708 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:59 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:35.738 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-18T07:17:35.738 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-18T07:17:35.739 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-18T07:17:35.739 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-18T07:17:35.739 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-18T07:17:35.739 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:36.133 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:36.259 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-fuse 2024-01-18T07:17:36.981 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:01:55 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:37.189 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:37.190 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:37.191 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-01-18T07:17:37.191 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:37.191 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 968 k 2024-01-18T07:17:37.191 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 3.2 M 2024-01-18T07:17:37.191 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:37.464 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 18 MB/s | 885 kB 00:00 2024-01-18T07:17:37.564 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 556 kB/s | 83 kB 00:00 2024-01-18T07:17:37.564 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:37.564 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-01-18T07:17:37.574 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:37.595 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:37.595 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:37.682 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:37.685 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:37.861 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:38.148 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-18T07:17:38.374 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T07:17:38.450 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-18T07:17:38.451 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout:Install 2 Packages 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 968 k 2024-01-18T07:17:38.452 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 3.2 M 2024-01-18T07:17:38.453 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:38.727 INFO:teuthology.orchestra.run.smithi106.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-01-18T07:17:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T07:17:38.757 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-18T07:17:38.827 INFO:teuthology.orchestra.run.smithi106.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 551 kB/s | 83 kB 00:00 2024-01-18T07:17:38.827 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:38.828 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-01-18T07:17:38.838 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:38.859 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:38.859 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:38.946 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:38.949 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:39.130 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:39.144 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-18T07:17:39.144 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:39.145 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:39.145 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-18T07:17:39.145 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:39.145 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:39.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-01-18T07:17:39.657 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-18T07:17:39.848 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T07:17:40.238 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-18T07:17:40.238 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-18T07:17:40.345 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:04 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:40.655 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-18T07:17:40.655 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:40.656 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:40.656 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-18T07:17:40.656 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:40.656 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:40.885 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados-devel 2024-01-18T07:17:41.616 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:00 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:41.823 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:41.823 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:41.823 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:41.823 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:17:41.824 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:41.825 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 131 k 2024-01-18T07:17:41.825 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 439 k 2024-01-18T07:17:41.825 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:41.869 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2024-01-18T07:17:41.869 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:41.869 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.8 MB/s | 131 kB 00:00 2024-01-18T07:17:41.870 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:41.875 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:41.875 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:41.924 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:41.927 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:42.039 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:42.288 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:42.584 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:42.975 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:42.975 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:42.975 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:42.976 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-18T07:17:42.976 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:42.976 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:43.083 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:43.083 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:17:43.084 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:43.085 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 131 k 2024-01-18T07:17:43.085 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 439 k 2024-01-18T07:17:43.085 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:43.133 INFO:teuthology.orchestra.run.smithi106.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-01-18T07:17:43.134 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:43.134 INFO:teuthology.orchestra.run.smithi106.stdout:Total 2.6 MB/s | 131 kB 00:00 2024-01-18T07:17:43.134 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:43.139 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:43.140 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:43.189 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:43.192 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:43.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-01-18T07:17:43.303 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:43.480 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:43.773 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:43.968 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:07 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:44.137 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:44.137 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:44.137 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:44.138 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-18T07:17:44.138 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:44.138 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:44.367 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs2 2024-01-18T07:17:45.099 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:04 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:45.407 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:45.453 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:45.454 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:17:45.454 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:46.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-01-18T07:17:46.549 INFO:teuthology.orchestra.run.smithi106.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:46.594 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:46.595 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:46.595 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:46.692 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install libcephfs-devel 2024-01-18T07:17:46.855 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:10 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:47.416 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:06 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:48.340 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:48.340 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:17:48.341 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:48.342 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 24 k 2024-01-18T07:17:48.342 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 94 k 2024-01-18T07:17:48.342 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:17:48.399 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 425 kB/s | 24 kB 00:00 2024-01-18T07:17:48.400 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:48.400 INFO:teuthology.orchestra.run.smithi084.stdout:Total 416 kB/s | 24 kB 00:00 2024-01-18T07:17:48.400 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:17:48.403 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:17:48.403 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:17:48.405 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:17:48.408 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:17:48.456 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:17:48.645 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:48.888 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:17:48.889 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:17:48.890 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:17:48.890 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:48.890 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 24 k 2024-01-18T07:17:48.890 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 94 k 2024-01-18T07:17:48.890 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:17:48.932 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:48.938 INFO:teuthology.orchestra.run.smithi106.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 514 kB/s | 24 kB 00:00 2024-01-18T07:17:48.938 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:17:48.939 INFO:teuthology.orchestra.run.smithi106.stdout:Total 497 kB/s | 24 kB 00:00 2024-01-18T07:17:48.939 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:17:48.942 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:17:48.943 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:17:48.945 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:17:48.948 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:17:48.994 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:17:49.171 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:49.284 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:49.284 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:49.284 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:17:49.284 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-18T07:17:49.285 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:17:49.285 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:49.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-01-18T07:17:49.457 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:17:49.803 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:49.907 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librados2 2024-01-18T07:17:50.169 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:14 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:50.626 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:09 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:51.605 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:51.680 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:51.681 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:17:51.681 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:52.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-01-18T07:17:52.066 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:52.111 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:52.112 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:52.112 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:52.214 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install librbd1 2024-01-18T07:17:52.752 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:16 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:52.934 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:11 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:54.188 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:54.233 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:54.234 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:17:54.234 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:54.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-01-18T07:17:54.353 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:54.399 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:54.400 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:54.400 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:54.491 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rados 2024-01-18T07:17:55.084 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:19 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:55.215 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:14 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:56.500 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:56.545 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:56.546 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:17:56.546 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:56.653 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:56.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-01-18T07:17:56.699 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:56.700 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:57.001 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rgw 2024-01-18T07:17:57.385 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:21 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:17:57.724 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:16 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:17:58.829 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:58.874 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:17:58.876 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:17:58.876 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:17:58.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-01-18T07:17:59.150 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:17:59.195 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:17:59.196 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:17:59.197 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:17:59.296 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-cephfs 2024-01-18T07:17:59.716 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:23 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:18:00.017 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:19 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:18:01.150 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:18:01.195 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:18:01.196 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:18:01.196 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:18:01.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-01-18T07:18:01.444 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:18:01.489 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:18:01.490 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:18:01.490 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:18:01.584 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install python3-rbd 2024-01-18T07:18:02.041 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:26 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:18:02.313 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:21 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:18:03.476 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:18:03.521 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:18:03.522 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-01-18T07:18:03.522 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:18:03.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-01-18T07:18:03.738 INFO:teuthology.orchestra.run.smithi106.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-18T07:18:03.783 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:18:03.785 INFO:teuthology.orchestra.run.smithi106.stdout:Nothing to do. 2024-01-18T07:18:03.785 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:18:03.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-fuse 2024-01-18T07:18:04.361 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:28 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:18:04.616 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:23 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:18:05.840 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:18:05.841 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:05.841 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:05.841 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:05.841 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:18:05.841 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 89 k 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 298 k 2024-01-18T07:18:05.842 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:18:05.896 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-01-18T07:18:05.896 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:05.897 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.6 MB/s | 89 kB 00:00 2024-01-18T07:18:05.897 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:18:05.906 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:18:05.906 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:18:05.955 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:18:05.958 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:18:06.063 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:18:06.108 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 89 k 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 298 k 2024-01-18T07:18:06.110 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:18:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2024-01-18T07:18:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-01-18T07:18:06.156 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:18:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:18:06.165 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:18:06.214 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:18:06.217 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:18:06.300 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:06.320 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:18:06.535 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:06.811 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:06.974 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:18:07.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-01-18T07:18:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:07.292 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:18:07.293 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:07.293 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:07.293 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:18:07.500 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-mirror 2024-01-18T07:18:07.943 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:31 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:18:08.238 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:27 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:18:09.420 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:18:09.421 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:09.422 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.7 M 2024-01-18T07:18:09.422 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 18 M 2024-01-18T07:18:09.422 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:18:09.515 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 40 MB/s | 3.7 MB 00:00 2024-01-18T07:18:09.516 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:09.516 INFO:teuthology.orchestra.run.smithi084.stdout:Total 39 MB/s | 3.7 MB 00:00 2024-01-18T07:18:09.517 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:18:09.528 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:18:09.528 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:18:09.590 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:18:09.593 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:18:09.735 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:09.736 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:18:09.737 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:09.737 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 3.7 M 2024-01-18T07:18:09.737 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 18 M 2024-01-18T07:18:09.737 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:18:09.817 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-01-18T07:18:09.818 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:09.818 INFO:teuthology.orchestra.run.smithi106.stdout:Total 46 MB/s | 3.7 MB 00:00 2024-01-18T07:18:09.818 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:18:09.830 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:18:09.830 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:18:09.892 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:18:09.895 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:18:10.063 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:18:10.267 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:10.376 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:18:10.561 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:10.686 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:10.977 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:11.139 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:18:11.343 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:11.344 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:11.344 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:18:11.344 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:11.344 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:11.344 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:18:11.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-01-18T07:18:11.516 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install rbd-nbd 2024-01-18T07:18:12.117 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:02:36 ago on Thu 18 Jan 2024 07:15:36 AM UTC. 2024-01-18T07:18:12.253 INFO:teuthology.orchestra.run.smithi106.stdout:Last metadata expiration check: 0:02:31 ago on Thu 18 Jan 2024 07:15:41 AM UTC. 2024-01-18T07:18:13.593 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-01-18T07:18:13.594 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:13.594 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:13.594 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-01-18T07:18:13.595 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:13.596 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 173 k 2024-01-18T07:18:13.596 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 649 k 2024-01-18T07:18:13.596 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-01-18T07:18:13.643 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2024-01-18T07:18:13.644 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:13.644 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.4 MB/s | 173 kB 00:00 2024-01-18T07:18:13.644 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-01-18T07:18:13.655 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-01-18T07:18:13.655 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-01-18T07:18:13.704 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-01-18T07:18:13.706 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-01-18T07:18:13.723 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-01-18T07:18:13.723 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:13.723 INFO:teuthology.orchestra.run.smithi106.stdout: Package Architecture Version Repository Size 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout:================================================================================ 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout:Install 1 Package 2024-01-18T07:18:13.724 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:13.725 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 173 k 2024-01-18T07:18:13.725 INFO:teuthology.orchestra.run.smithi106.stdout:Installed size: 649 k 2024-01-18T07:18:13.725 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-01-18T07:18:13.770 INFO:teuthology.orchestra.run.smithi106.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2024-01-18T07:18:13.770 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-01-18T07:18:13.771 INFO:teuthology.orchestra.run.smithi106.stdout:Total 3.6 MB/s | 173 kB 00:00 2024-01-18T07:18:13.771 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-01-18T07:18:13.781 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-01-18T07:18:13.782 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-01-18T07:18:13.820 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-01-18T07:18:13.832 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-01-18T07:18:13.835 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-01-18T07:18:13.954 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-01-18T07:18:14.023 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.127 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.350 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.403 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.738 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.738 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:14.738 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-01-18T07:18:14.738 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:14.738 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:18:14.739 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:18:14.785 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-01-18T07:18:14.980 DEBUG:teuthology.parallel:result is None 2024-01-18T07:18:15.016 DEBUG:teuthology.parallel:result is None 2024-01-18T07:18:15.017 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:18:15.017 INFO:teuthology.packaging:ref: None 2024-01-18T07:18:15.017 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:18:15.017 INFO:teuthology.packaging:branch: None 2024-01-18T07:18:15.017 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:18:15.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T07:18:15.247 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-18T07:18:15.270 INFO:teuthology.orchestra.run.smithi084.stdout:16.2.4-0.el8 2024-01-18T07:18:15.271 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-18T07:18:15.271 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-18T07:18:15.273 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-18T07:18:15.273 INFO:teuthology.packaging:ref: None 2024-01-18T07:18:15.273 INFO:teuthology.packaging:tag: v16.2.4 2024-01-18T07:18:15.273 INFO:teuthology.packaging:branch: None 2024-01-18T07:18:15.273 INFO:teuthology.packaging:sha1: 491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0 2024-01-18T07:18:15.273 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-18T07:18:15.434 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-18T07:18:15.458 INFO:teuthology.orchestra.run.smithi106.stdout:16.2.4-0.el8 2024-01-18T07:18:15.459 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-18T07:18:15.459 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-18T07:18:15.461 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-18T07:18:15.461 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:18:15.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-18T07:18:15.489 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:18:15.489 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-18T07:18:15.528 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-18T07:18:15.528 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:18:15.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-01-18T07:18:15.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-18T07:18:15.622 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:18:15.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-01-18T07:18:15.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-18T07:18:15.716 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-18T07:18:15.717 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:18:15.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-18T07:18:15.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-18T07:18:15.809 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:18:15.809 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-18T07:18:15.836 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-18T07:18:15.902 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-18T07:18:15.902 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:18:15.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-01-18T07:18:15.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-18T07:18:15.996 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:18:15.996 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-01-18T07:18:16.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-18T07:18:16.089 INFO:teuthology.run_tasks:Running task print... 2024-01-18T07:18:16.123 INFO:teuthology.task.print:**** done install task... 2024-01-18T07:18:16.124 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-18T07:18:16.201 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0'} 2024-01-18T07:18:16.201 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-18T07:18:16.202 INFO:tasks.cephadm:Cluster fsid is bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:16.202 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-18T07:18:16.202 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-18T07:18:16.202 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi084': '172.21.15.84', 'mon.smithi106': '172.21.15.106'} 2024-01-18T07:18:16.202 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-18T07:18:16.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-01-18T07:18:16.230 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-01-18T07:18:16.257 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-18T07:18:16.257 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.504 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 18 07:18 /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.504 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.710 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 18 07:18 /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.710 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.729 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-18T07:18:16.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-01-18T07:18:16.796 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-01-18T07:18:16.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-01-18T07:18:16.864 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-01-18T07:18:16.890 INFO:tasks.cephadm:Writing seed config... 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-18T07:18:16.891 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-18T07:18:16.892 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-18T07:18:16.893 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-18T07:18:16.894 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:18:16.894 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-18T07:18:16.922 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bf9448d2-b5d1-11ee-95ae-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-18T07:18:16.923 DEBUG:teuthology.orchestra.run.smithi084:mon.smithi084> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service 2024-01-18T07:18:16.965 INFO:tasks.cephadm:Bootstrapping... 2024-01-18T07:18:16.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.84 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:18:16.991 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:-- Logs begin at Thu 2024-01-18 07:09:07 UTC. -- 2024-01-18T07:18:17.248 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T07:18:18.242 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.6.1 2024-01-18T07:18:18.249 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying podman|docker is present... 2024-01-18T07:18:18.249 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T07:18:18.272 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.6.1 2024-01-18T07:18:18.280 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying lvm2 is present... 2024-01-18T07:18:18.280 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying time synchronization is in place... 2024-01-18T07:18:18.281 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chrony.service 2024-01-18T07:18:18.288 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-18T07:18:18.289 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chrony.service 2024-01-18T07:18:18.294 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:18:18.295 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-18T07:18:18.301 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-01-18T07:18:18.301 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chronyd.service 2024-01-18T07:18:18.307 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-01-18T07:18:18.308 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-01-18T07:18:18.308 INFO:teuthology.orchestra.run.smithi084.stderr:Repeating the final host check... 2024-01-18T07:18:18.308 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-18T07:18:18.328 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 4.6.1 2024-01-18T07:18:18.337 INFO:teuthology.orchestra.run.smithi084.stderr:podman|docker (/bin/podman) is present 2024-01-18T07:18:18.337 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl is present 2024-01-18T07:18:18.337 INFO:teuthology.orchestra.run.smithi084.stderr:lvcreate is present 2024-01-18T07:18:18.337 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chrony.service 2024-01-18T07:18:18.343 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-18T07:18:18.343 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chrony.service 2024-01-18T07:18:18.348 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:18:18.349 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-18T07:18:18.355 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: enabled 2024-01-18T07:18:18.355 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active chronyd.service 2024-01-18T07:18:18.360 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: active 2024-01-18T07:18:18.361 INFO:teuthology.orchestra.run.smithi084.stderr:Unit chronyd.service is enabled and running 2024-01-18T07:18:18.361 INFO:teuthology.orchestra.run.smithi084.stderr:Host looks OK 2024-01-18T07:18:18.361 INFO:teuthology.orchestra.run.smithi084.stderr:Cluster fsid: bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:18.361 INFO:teuthology.orchestra.run.smithi084.stderr:Acquiring lock 140602714339816 on /run/cephadm/bf9448d2-b5d1-11ee-95ae-87774f69a715.lock 2024-01-18T07:18:18.361 INFO:teuthology.orchestra.run.smithi084.stderr:Lock 140602714339816 acquired on /run/cephadm/bf9448d2-b5d1-11ee-95ae-87774f69a715.lock 2024-01-18T07:18:18.362 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 3300 ... 2024-01-18T07:18:18.362 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying IP 172.21.15.84 port 6789 ... 2024-01-18T07:18:18.362 INFO:teuthology.orchestra.run.smithi084.stderr:Base mon IP is 172.21.15.84, final addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-01-18T07:18:18.362 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip route ls 2024-01-18T07:18:18.365 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-01-18T07:18:18.365 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-01-18T07:18:18.365 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip -6 route ls 2024-01-18T07:18:18.368 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-18T07:18:18.368 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-18T07:18:18.368 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1747sec hoplimit 64 pref medium 2024-01-18T07:18:18.368 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /sbin/ip -6 addr ls 2024-01-18T07:18:18.370 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-18T07:18:18.371 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-18T07:18:18.371 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-18T07:18:18.371 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-18T07:18:18.371 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d751/64 scope link 2024-01-18T07:18:18.371 INFO:teuthology.orchestra.run.smithi084.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-18T07:18:18.372 INFO:teuthology.orchestra.run.smithi084.stderr:Mon IP 172.21.15.84 is in CIDR network 172.21.0.0/20 2024-01-18T07:18:18.372 INFO:teuthology.orchestra.run.smithi084.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-18T07:18:18.373 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-18T07:18:18.373 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-18T07:18:18.467 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-18T07:18:25.032 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Getting image source signatures 2024-01-18T07:18:25.043 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-18T07:18:25.043 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-18T07:18:36.707 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-18T07:18:36.765 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: Writing manifest to image destination 2024-01-18T07:18:37.038 INFO:teuthology.orchestra.run.smithi084.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-18T07:18:37.052 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-18T07:18:37.581 INFO:teuthology.orchestra.run.smithi084.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-18T07:18:37.946 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-18T07:18:37.947 INFO:teuthology.orchestra.run.smithi084.stderr:Extracting ceph user uid/gid from container image... 2024-01-18T07:18:37.947 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-18T07:18:38.203 INFO:teuthology.orchestra.run.smithi084.stderr:stat: 167 167 2024-01-18T07:18:38.536 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial keys... 2024-01-18T07:18:38.536 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T07:18:38.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQDO0Khl9OSdLxAARn5y4f+mGyF5tJ1fw5vplw== 2024-01-18T07:18:39.151 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T07:18:39.445 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQDP0KhlKUOAGRAA0nsbcZD8pG3p8gsAyxd5Vw== 2024-01-18T07:18:39.770 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-18T07:18:40.086 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-authtool: AQDQ0Khlj9DcAxAAU/7AH9HGQzRDFKIcc/F9+w== 2024-01-18T07:18:40.424 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial monmap... 2024-01-18T07:18:40.424 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpvwc5lqbu:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --addv smithi084 [v2:172.21.15.84:3300,v1:172.21.15.84:6789] /tmp/monmap 2024-01-18T07:18:40.728 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-18T07:18:40.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:40.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-18T07:18:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:monmaptool for smithi084 [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-18T07:18:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: set fsid to bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-18T07:18:41.065 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:18:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mon... 2024-01-18T07:18:41.066 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z -v /tmp/ceph-tmpernrxoln:/tmp/keyring:z -v /tmp/ceph-tmpvwc5lqbu:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi084 --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-18T07:18:41.364 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.327+0000 7f42565c2700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-18T07:18:41.364 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.327+0000 7f42565c2700 1 imported monmap: 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: last_changed 2024-01-18T07:18:40.708288+0000 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: created 2024-01-18T07:18:40.708288+0000 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.smithi084 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.327+0000 7f42565c2700 0 /usr/bin/ceph-mon: set fsid to bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: RocksDB version: 6.8.1 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Compile date May 13 2021 2024-01-18T07:18:41.365 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: DB SUMMARY 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 0, files: 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.error_if_exists: 0 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.create_if_missing: 1 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.paranoid_checks: 1 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.env: 0x55cb9fd171c0 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.fs: Posix File System 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.info_log: 0x55cba0da4ec0 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.statistics: (nil) 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.use_fsync: 0 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_log_file_size: 0 2024-01-18T07:18:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.allow_fallocate: 1 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.use_direct_reads: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.db_log_dir: 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_subcompactions: 1 2024-01-18T07:18:41.367 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_background_flushes: -1 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.write_buffer_manager: 0x55cba0dab530 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-18T07:18:41.368 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.unordered_write: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.row_cache: None 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.wal_filter: None 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.preserve_deletes: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.two_write_queues: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-18T07:18:41.369 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.atomic_flush: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.log_readahead_size: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_background_jobs: 2 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_background_compactions: -1 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-18T07:18:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.max_open_files: -1 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Compression algorithms supported: 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kZSTD supported: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kXpressCompression supported: 0 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kLZ4Compression supported: 1 2024-01-18T07:18:41.371 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kBZip2Compression supported: 0 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kZlibCompression supported: 1 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: kSnappyCompression supported: 1 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.328+0000 7f42565c2700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.merge_operator: 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_filter: None 2024-01-18T07:18:41.372 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cba0cbc120) 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-18T07:18:41.373 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache: 0x55cba0cf2d10 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-18T07:18:41.375 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-18T07:18:41.376 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression: NoCompression 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-18T07:18:41.377 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.num_levels: 7 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-18T07:18:41.378 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-18T07:18:41.379 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-18T07:18:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.table_properties_collectors: 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.inplace_update_support: 0 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-18T07:18:41.381 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.bloom_locality: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.max_successive_merges: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.ttl: 2592000 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.329+0000 7f42565c2700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.330+0000 7f42565c2700 4 rocksdb: DB pointer 0x55cba0db7800 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.330+0000 7f423f45b700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-18T07:18:41.382 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.330+0000 7f423f45b700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.383 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.331+0000 7f42565c2700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.331+0000 7f42565c2700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-18T07:18:41.386 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph-mon: debug 2024-01-18T07:18:41.331+0000 7f42565c2700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi084 for mon.smithi084 2024-01-18T07:18:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:create mon.smithi084 on 2024-01-18T07:18:41.690 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:41.694 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl disable ceph.target 2024-01-18T07:18:41.701 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-18T07:18:41.788 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph.target 2024-01-18T07:18:41.794 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-18T07:18:41.876 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph.target 2024-01-18T07:18:41.882 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target 2024-01-18T07:18:41.887 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target. 2024-01-18T07:18:41.888 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target. 2024-01-18T07:18:41.963 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target 2024-01-18T07:18:41.970 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl daemon-reload 2024-01-18T07:18:42.056 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:18:42.078 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl reset-failed ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:18:42.084 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service: Unit ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service not loaded. 2024-01-18T07:18:42.084 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:18:42.091 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target.wants/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service → /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@.service. 2024-01-18T07:18:42.175 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:18:42.495 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:42 smithi084 bash[38312]: f63e038cd76c20c8b77b6b885b29cb871bfef31fc623d8463d1a9b102654a4a8 2024-01-18T07:18:42.498 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T07:18:42.507 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-18T07:18:42.508 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T07:18:42.515 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon to start... 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mon... 2024-01-18T07:18:42.517 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-18T07:18:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:42 smithi084 systemd[1]: Started Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715. 2024-01-18T07:18:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.511562+0000 mon.smithi084 (mon.0) 0 : [INF] mkfs bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:42.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.505438+0000 mon.smithi084 (mon.0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-18T07:18:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.956+0000 7f4229baa700 1 Processor -- start 2024-01-18T07:18:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.956+0000 7f4229baa700 1 -- start start 2024-01-18T07:18:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f4229baa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:42.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f4229baa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4224072890 con 0x7f4224072470 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f42237fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f42237fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52238/0 (socket says 172.21.15.84:52238) 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f42237fe700 1 -- 172.21.15.84:0/2255715259 learned_addr learned my addr 172.21.15.84:0/2255715259 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.957+0000 7f42237fe700 1 -- 172.21.15.84:0/2255715259 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224070f00 con 0x7f4224072470 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f42237fe700 1 --2- 172.21.15.84:0/2255715259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f421800b5f0 tx=0x7f4218006ee0).ready entity=mon.0 client_cookie=95269a1cde07c7eb server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f42227fc700 1 -- 172.21.15.84:0/2255715259 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421800d600 con 0x7f4224072470 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f42227fc700 1 -- 172.21.15.84:0/2255715259 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421800d760 con 0x7f4224072470 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f42227fc700 1 -- 172.21.15.84:0/2255715259 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421800da30 con 0x7f4224072470 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 -- 172.21.15.84:0/2255715259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 msgr2=0x7f4224070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 --2- 172.21.15.84:0/2255715259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f421800b5f0 tx=0x7f4218006ee0).stop 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 -- 172.21.15.84:0/2255715259 shutdown_connections 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 --2- 172.21.15.84:0/2255715259 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f4224070a10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:42.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 -- 172.21.15.84:0/2255715259 >> 172.21.15.84:0/2255715259 conn(0x7f42240f74c0 msgr2=0x7f42240f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.958+0000 7f4229baa700 1 -- 172.21.15.84:0/2255715259 shutdown_connections 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f4229baa700 1 -- 172.21.15.84:0/2255715259 wait complete. 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f4229baa700 1 Processor -- start 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f4229baa700 1 -- start start 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f4229baa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f4229baa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422410cbd0 con 0x7f4224072470 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f42237fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f42237fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52240/0 (socket says 172.21.15.84:52240) 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.959+0000 7f42237fe700 1 -- 172.21.15.84:0/1075662971 learned_addr learned my addr 172.21.15.84:0/1075662971 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f42237fe700 1 -- 172.21.15.84:0/1075662971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421800b200 con 0x7f4224072470 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f42237fe700 1 --2- 172.21.15.84:0/1075662971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4218006c70 tx=0x7f4218016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4218011b40 con 0x7f4224072470 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218011ca0 con 0x7f4224072470 2024-01-18T07:18:42.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421801d430 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422410cdd0 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422410d1e0 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f421800f070 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.960+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4218027460 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.961+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4224059100 con 0x7f4224072470 2024-01-18T07:18:42.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:42.964+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4218016070 con 0x7f4224072470 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.029+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4224071830 con 0x7f4224072470 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f420bfff700 1 -- 172.21.15.84:0/1075662971 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+323 (secure 0 0 0) 0x7f4218036070 con 0x7f4224072470 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: cluster: 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: id: bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: services: 2024-01-18T07:18:43.033 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi084 (age 0.5204s) 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: data: 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: pgs: 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 msgr2=0x7f422410c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 --2- 172.21.15.84:0/1075662971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4218006c70 tx=0x7f4218016000).stop 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 shutdown_connections 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 --2- 172.21.15.84:0/1075662971 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4224072470 0x7f422410c6e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 >> 172.21.15.84:0/1075662971 conn(0x7f42240f74c0 msgr2=0x7f4224100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:43.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 shutdown_connections 2024-01-18T07:18:43.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.030+0000 7f4229baa700 1 -- 172.21.15.84:0/1075662971 wait complete. 2024-01-18T07:18:43.388 INFO:teuthology.orchestra.run.smithi084.stderr:mon is available 2024-01-18T07:18:43.389 INFO:teuthology.orchestra.run.smithi084.stderr:Assimilating anything we can from ceph.conf... 2024-01-18T07:18:43.389 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi084/config 2024-01-18T07:18:43.585 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.510336+0000 mon.smithi084 (mon.0) 2 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-18T07:18:43.585 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.510530+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} 2024-01-18T07:18:43.586 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.513808+0000 mon.smithi084 (mon.0) 4 : cluster [DBG] fsmap 2024-01-18T07:18:43.586 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.515830+0000 mon.smithi084 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-18T07:18:43.586 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: cluster 2024-01-18T07:18:42.516306+0000 mon.smithi084 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-18T07:18:43.586 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: audit 2024-01-18T07:18:43.030686+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/1075662971' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-18T07:18:43.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01bb0b4700 1 Processor -- start 2024-01-18T07:18:43.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01bb0b4700 1 -- start start 2024-01-18T07:18:43.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01bb0b4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:43.840 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01bb0b4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b40fee10 con 0x7f01b40703f0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01b8e50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.838+0000 7f01b8e50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52244/0 (socket says 172.21.15.84:52244) 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b8e50700 1 -- 172.21.15.84:0/2588763876 learned_addr learned my addr 172.21.15.84:0/2588763876 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b8e50700 1 -- 172.21.15.84:0/2588763876 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b40fef50 con 0x7f01b40703f0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b8e50700 1 --2- 172.21.15.84:0/2588763876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f01a800b5f0 tx=0x7f01a8006ee0).ready entity=mon.0 client_cookie=78abde0ed451a15c server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b37fe700 1 -- 172.21.15.84:0/2588763876 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a800d600 con 0x7f01b40703f0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b37fe700 1 -- 172.21.15.84:0/2588763876 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a800d760 con 0x7f01b40703f0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01b37fe700 1 -- 172.21.15.84:0/2588763876 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a800da30 con 0x7f01b40703f0 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2588763876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 msgr2=0x7f01b40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.839+0000 7f01bb0b4700 1 --2- 172.21.15.84:0/2588763876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f01a800b5f0 tx=0x7f01a8006ee0).stop 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2588763876 shutdown_connections 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 --2- 172.21.15.84:0/2588763876 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b40fe920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:43.841 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2588763876 >> 172.21.15.84:0/2588763876 conn(0x7f01b40f63c0 msgr2=0x7f01b40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:43.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2588763876 shutdown_connections 2024-01-18T07:18:43.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2588763876 wait complete. 2024-01-18T07:18:43.843 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 Processor -- start 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.840+0000 7f01bb0b4700 1 -- start start 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01bb0b4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01bb0b4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b410a770 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b8e50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b8e50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52260/0 (socket says 172.21.15.84:52260) 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b8e50700 1 -- 172.21.15.84:0/2792124969 learned_addr learned my addr 172.21.15.84:0/2792124969 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b8e50700 1 -- 172.21.15.84:0/2792124969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a800b200 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b8e50700 1 --2- 172.21.15.84:0/2792124969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f01a8006c70 tx=0x7f01a8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a8011b40 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a8011ca0 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b410a970 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01a801d430 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b410ad80 con 0x7f01b40703f0 2024-01-18T07:18:43.844 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f01a800f070 con 0x7f01b40703f0 2024-01-18T07:18:43.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.841+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f01a8027460 con 0x7f01b40703f0 2024-01-18T07:18:43.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.842+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b4045ad0 con 0x7f01b40703f0 2024-01-18T07:18:43.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.844+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f01a8016070 con 0x7f01b40703f0 2024-01-18T07:18:43.910 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.909+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f01b40ff1a0 con 0x7f01b40703f0 2024-01-18T07:18:43.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.912+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01a801d5e0 con 0x7f01b40703f0 2024-01-18T07:18:43.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.912+0000 7f01b17fa700 1 -- 172.21.15.84:0/2792124969 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f01a8036020 con 0x7f01b40703f0 2024-01-18T07:18:43.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [global] 2024-01-18T07:18:43.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: fsid = bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [mgr] 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: [osd] 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-18T07:18:43.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 msgr2=0x7f01b410a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 --2- 172.21.15.84:0/2792124969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f01a8006c70 tx=0x7f01a8016000).stop 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 shutdown_connections 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 --2- 172.21.15.84:0/2792124969 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f01b40703f0 0x7f01b410a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 >> 172.21.15.84:0/2792124969 conn(0x7f01b40f63c0 msgr2=0x7f01b40f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 shutdown_connections 2024-01-18T07:18:43.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:43.913+0000 7f01bb0b4700 1 -- 172.21.15.84:0/2792124969 wait complete. 2024-01-18T07:18:44.263 INFO:teuthology.orchestra.run.smithi084.stderr:Generating new minimal ceph.conf... 2024-01-18T07:18:44.264 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z -v /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084:/var/lib/ceph/mon/ceph-smithi084:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi084/config 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08ebb8700 1 Processor -- start 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08ebb8700 1 -- start start 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08ebb8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08ebb8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff088072980 con 0x7ff088072560 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08c954700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08c954700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52262/0 (socket says 172.21.15.84:52262) 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08c954700 1 -- 172.21.15.84:0/3486915398 learned_addr learned my addr 172.21.15.84:0/3486915398 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08c954700 1 -- 172.21.15.84:0/3486915398 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088070ff0 con 0x7ff088072560 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.666+0000 7ff08c954700 1 --2- 172.21.15.84:0/3486915398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff07c00b5f0 tx=0x7ff07c006ee0).ready entity=mon.0 client_cookie=e8ad328062ec1b73 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:44.670 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff0877fe700 1 -- 172.21.15.84:0/3486915398 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07c00d600 con 0x7ff088072560 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff0877fe700 1 -- 172.21.15.84:0/3486915398 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff07c00dbd0 con 0x7ff088072560 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff0877fe700 1 -- 172.21.15.84:0/3486915398 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07c011c00 con 0x7ff088072560 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3486915398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 msgr2=0x7ff088070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 --2- 172.21.15.84:0/3486915398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff07c00b5f0 tx=0x7ff07c006ee0).stop 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3486915398 shutdown_connections 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 --2- 172.21.15.84:0/3486915398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff088070b00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3486915398 >> 172.21.15.84:0/3486915398 conn(0x7ff0880f74b0 msgr2=0x7ff0880f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3486915398 shutdown_connections 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.667+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3486915398 wait complete. 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08ebb8700 1 Processor -- start 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08ebb8700 1 -- start start 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08ebb8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:44.671 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08ebb8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08810d220 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08c954700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08c954700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52278/0 (socket says 172.21.15.84:52278) 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08c954700 1 -- 172.21.15.84:0/3233673475 learned_addr learned my addr 172.21.15.84:0/3233673475 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08c954700 1 -- 172.21.15.84:0/3233673475 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c00b200 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.668+0000 7ff08c954700 1 --2- 172.21.15.84:0/3233673475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff07c006c70 tx=0x7ff07c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07c00da30 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff07c01a9f0 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08810d420 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff07c01b890 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08810d830 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff07c00f070 con 0x7ff088072560 2024-01-18T07:18:44.672 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.669+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff07c025af0 con 0x7ff088072560 2024-01-18T07:18:44.673 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.670+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff088045ad0 con 0x7ff088072560 2024-01-18T07:18:44.673 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.672+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff07c016070 con 0x7ff088072560 2024-01-18T07:18:44.736 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.735+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff088101df0 con 0x7ff088072560 2024-01-18T07:18:44.736 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff0857fa700 1 -- 172.21.15.84:0/3233673475 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff07c01bd90 con 0x7ff088072560 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 msgr2=0x7ff08810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 --2- 172.21.15.84:0/3233673475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff07c006c70 tx=0x7ff07c006ee0).stop 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 shutdown_connections 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 --2- 172.21.15.84:0/3233673475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff088072560 0x7ff08810cd30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 >> 172.21.15.84:0/3233673475 conn(0x7ff0880f74b0 msgr2=0x7ff0880f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:44.737 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 shutdown_connections 2024-01-18T07:18:44.738 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:44.736+0000 7ff08ebb8700 1 -- 172.21.15.84:0/3233673475 wait complete. 2024-01-18T07:18:44.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: audit 2024-01-18T07:18:43.910460+0000 mon.smithi084 (mon.0) 8 : audit [INF] from='client.? 172.21.15.84:0/2792124969' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-18T07:18:44.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: audit 2024-01-18T07:18:43.913432+0000 mon.smithi084 (mon.0) 9 : audit [INF] from='client.? 172.21.15.84:0/2792124969' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-18T07:18:45.105 INFO:teuthology.orchestra.run.smithi084.stderr:Restarting the monitor... 2024-01-18T07:18:45.105 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:18:45.186 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:18:45.437 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: debug 2024-01-18T07:18:45.185+0000 7f0209b30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-18T07:18:45.437 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38327]: debug 2024-01-18T07:18:45.185+0000 7f0209b30700 -1 mon.smithi084@0(leader) e1 *** Got Signal Terminated *** 2024-01-18T07:18:45.438 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 podman[38651]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084 2024-01-18T07:18:45.837 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 systemd[1]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service: Succeeded. 2024-01-18T07:18:45.838 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 systemd[1]: Stopped Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715. 2024-01-18T07:18:45.838 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 systemd[1]: Starting Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:18:45.847 INFO:teuthology.orchestra.run.smithi084.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-18T07:18:45.848 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-18T07:18:46.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.836+0000 7fbf537dc700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-18T07:18:46.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.837+0000 7fbf537dc700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-18T07:18:46.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.837+0000 7fbf537dc700 0 pidfile_write: ignore empty --pid-file 2024-01-18T07:18:46.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.841+0000 7fbf537dc700 0 load: jerasure load: lrc load: isa 2024-01-18T07:18:46.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: RocksDB version: 6.8.1 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Compile date May 13 2021 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: DB SUMMARY 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: CURRENT file: CURRENT 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi084/store.db dir, Total Num: 1, files: 000004.sst 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi084/store.db: 000006.log size: 92918 ; 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.error_if_exists: 0 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.create_if_missing: 0 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.paranoid_checks: 1 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.env: 0x560c4ed931c0 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.fs: Posix File System 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.info_log: 0x560c50533960 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.statistics: (nil) 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.use_fsync: 0 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_log_file_size: 0 2024-01-18T07:18:46.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 bash[38722]: 0d19a9959a298009da722792f94381c867014af242c6e3bd55780a634799482d 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.allow_fallocate: 1 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.use_direct_reads: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.db_log_dir: 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi084/store.db 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-18T07:18:46.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_subcompactions: 1 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_background_flushes: -1 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.write_buffer_manager: 0x560c511a8030 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-18T07:18:46.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.unordered_write: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.row_cache: None 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.wal_filter: None 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.preserve_deletes: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.two_write_queues: 0 2024-01-18T07:18:46.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.atomic_flush: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.log_readahead_size: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_background_jobs: 2 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_background_compactions: -1 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-18T07:18:46.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_open_files: -1 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-18T07:18:46.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Compression algorithms supported: 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kZSTD supported: 0 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kXpressCompression supported: 0 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kLZ4Compression supported: 1 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kBZip2Compression supported: 0 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kZlibCompression supported: 1 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: kSnappyCompression supported: 1 2024-01-18T07:18:46.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.merge_operator: 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_filter: None 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560c5044a128) 2024-01-18T07:18:46.143 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cache_index_and_filter_blocks: 1 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: pin_top_level_index_and_filter: 1 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: index_type: 0 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: data_block_index_type: 0 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: index_shortening: 1 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: data_block_hash_table_util_ratio: 0.750000 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: hash_index_allow_collision: 1 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: checksum: 1 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: no_block_cache: 0 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_cache: 0x560c50480f10 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_cache_name: BinnedLRUCache 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_cache_options: 2024-01-18T07:18:46.144 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: capacity : 536870912 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: num_shard_bits : 4 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: strict_capacity_limit : 0 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: high_pri_pool_ratio: 0.000 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_cache_compressed: (nil) 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: persistent_cache: (nil) 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_size: 4096 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_size_deviation: 10 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_restart_interval: 16 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: index_block_restart_interval: 1 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: metadata_block_size: 4096 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: partition_filters: 0 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: use_delta_encoding: 1 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: whole_key_filtering: 1 2024-01-18T07:18:46.145 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: verify_compression: 0 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: read_amp_bytes_per_bit: 0 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: format_version: 2 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: enable_index_compression: 1 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: block_align: 0 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression: NoCompression 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.num_levels: 7 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-18T07:18:46.146 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-18T07:18:46.147 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-18T07:18:46.148 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-18T07:18:46.149 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.table_properties_collectors: 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.inplace_update_support: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.bloom_locality: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.max_successive_merges: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.ttl: 2592000 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-18T07:18:46.150 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi084/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.842+0000 7fbf537dc700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.843+0000 7fbf537dc700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562325843862, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.843+0000 7fbf537dc700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.843+0000 7fbf537dc700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.845+0000 7fbf537dc700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562325845787, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705562325, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.845+0000 7fbf537dc700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.845+0000 7fbf537dc700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562325846246, "job": 1, "event": "recovery_finished"} 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 systemd[1]: Started Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715. 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.846+0000 7fbf537dc700 4 rocksdb: DB pointer 0x560c50545800 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.846+0000 7fbf3a5f4700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.846+0000 7fbf3a5f4700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-18T07:18:46.151 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** DB Stats ** 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** Compaction Stats [default] ** 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.152 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** Compaction Stats [default] ** 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Cumulative compaction: 0.00 GB write, 20.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Interval compaction: 0.00 GB write, 20.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.153 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** Compaction Stats [default] ** 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** Compaction Stats [default] ** 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 52.1 0.00 0.00 1 0.002 0 0 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:46.154 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Cumulative compaction: 0.00 GB write, 20.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.847+0000 7fbf537dc700 0 starting mon.smithi084 rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi084 fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 1 mon.smithi084@-1(???) e1 preinit fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).mds e1 new map 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).mds e1 print_map 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: e1 2024-01-18T07:18:46.155 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: legacy client fscid: -1 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: No filesystems configured 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 0 mon.smithi084@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:43.910460+0000 mon.smithi084 (mon.0) 8 : audit [INF] from='client.? 172.21.15.84:0/2792124969' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:43.913432+0000 mon.smithi084 (mon.0) 9 : audit [INF] from='client.? 172.21.15.84:0/2792124969' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-18T07:18:46.156 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:18:45.848+0000 7fbf537dc700 1 mon.smithi084@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-18T07:18:46.157 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:45.864596+0000 mon.smithi084 (mon.0) 1 : cluster [INF] mon.smithi084 is new leader, mons smithi084 in quorum (ranks 0) 2024-01-18T07:18:46.157 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:45.864657+0000 mon.smithi084 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]} 2024-01-18T07:18:46.157 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:45.864694+0000 mon.smithi084 (mon.0) 3 : cluster [DBG] fsmap 2024-01-18T07:18:46.157 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:45.864721+0000 mon.smithi084 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-18T07:18:46.157 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:45.865171+0000 mon.smithi084 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-18T07:18:46.229 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.227+0000 7fb6159d3700 1 Processor -- start 2024-01-18T07:18:46.229 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.227+0000 7fb6159d3700 1 -- start start 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.228+0000 7fb6159d3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.228+0000 7fb6159d3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100fb490 con 0x7fb6100fab80 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.228+0000 7fb60effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.228+0000 7fb60effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52288/0 (socket says 172.21.15.84:52288) 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.228+0000 7fb60effd700 1 -- 172.21.15.84:0/1350280912 learned_addr learned my addr 172.21.15.84:0/1350280912 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb60effd700 1 -- 172.21.15.84:0/1350280912 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6100fb5d0 con 0x7fb6100fab80 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb60effd700 1 --2- 172.21.15.84:0/1350280912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb60400b5f0 tx=0x7fb604006ee0).ready entity=mon.0 client_cookie=6ce8c64598c0988c server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb60dffb700 1 -- 172.21.15.84:0/1350280912 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60400d600 con 0x7fb6100fab80 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb60dffb700 1 -- 172.21.15.84:0/1350280912 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb60400dbd0 con 0x7fb6100fab80 2024-01-18T07:18:46.230 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb60dffb700 1 -- 172.21.15.84:0/1350280912 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb604011c00 con 0x7fb6100fab80 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb6159d3700 1 -- 172.21.15.84:0/1350280912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 msgr2=0x7fb6100fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb6159d3700 1 --2- 172.21.15.84:0/1350280912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb60400b5f0 tx=0x7fb604006ee0).stop 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb6159d3700 1 -- 172.21.15.84:0/1350280912 shutdown_connections 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb6159d3700 1 --2- 172.21.15.84:0/1350280912 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb6100fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.229+0000 7fb6159d3700 1 -- 172.21.15.84:0/1350280912 >> 172.21.15.84:0/1350280912 conn(0x7fb6100f63c0 msgr2=0x7fb6100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.230+0000 7fb6159d3700 1 -- 172.21.15.84:0/1350280912 shutdown_connections 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.230+0000 7fb6159d3700 1 -- 172.21.15.84:0/1350280912 wait complete. 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb6159d3700 1 Processor -- start 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb6159d3700 1 -- start start 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb6159d3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:46.232 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb6159d3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6100ff290 con 0x7fb6100fab80 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb60effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb60effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52298/0 (socket says 172.21.15.84:52298) 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb60effd700 1 -- 172.21.15.84:0/1691222189 learned_addr learned my addr 172.21.15.84:0/1691222189 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb60effd700 1 -- 172.21.15.84:0/1691222189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60400b200 con 0x7fb6100fab80 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb60effd700 1 --2- 172.21.15.84:0/1691222189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb60400b1d0 tx=0x7fb604006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60401b410 con 0x7fb6100fab80 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb60401b9e0 con 0x7fb6100fab80 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60401a840 con 0x7fb6100fab80 2024-01-18T07:18:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.231+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6100ff430 con 0x7fb6100fab80 2024-01-18T07:18:46.234 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.232+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6100ff840 con 0x7fb6100fab80 2024-01-18T07:18:46.234 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.232+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb60400f070 con 0x7fb6100fab80 2024-01-18T07:18:46.234 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.232+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb604025c90 con 0x7fb6100fab80 2024-01-18T07:18:46.234 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.232+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb610045ad0 con 0x7fb6100fab80 2024-01-18T07:18:46.235 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.235+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb604016070 con 0x7fb6100fab80 2024-01-18T07:18:46.298 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.297+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb6100fc890 con 0x7fb6100fab80 2024-01-18T07:18:46.303 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.302+0000 7fb5f77fe700 1 -- 172.21.15.84:0/1691222189 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb60401aea0 con 0x7fb6100fab80 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 msgr2=0x7fb610100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 --2- 172.21.15.84:0/1691222189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb60400b1d0 tx=0x7fb604006ee0).stop 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 shutdown_connections 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 --2- 172.21.15.84:0/1691222189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb6100fab80 0x7fb610100e10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 >> 172.21.15.84:0/1691222189 conn(0x7fb6100f63c0 msgr2=0x7fb610070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 shutdown_connections 2024-01-18T07:18:46.304 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:46.303+0000 7fb6159d3700 1 -- 172.21.15.84:0/1691222189 wait complete. 2024-01-18T07:18:46.663 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-18T07:18:46.663 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:18:46.663 INFO:teuthology.orchestra.run.smithi084.stderr:Creating mgr... 2024-01-18T07:18:46.663 INFO:teuthology.orchestra.run.smithi084.stderr:Verifying port 9283 ... 2024-01-18T07:18:46.665 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl daemon-reload 2024-01-18T07:18:46.756 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg 2024-01-18T07:18:46.785 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl reset-failed ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg 2024-01-18T07:18:46.791 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Failed to reset failed state of unit ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg.service: Unit ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg.service not loaded. 2024-01-18T07:18:46.791 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl enable ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg 2024-01-18T07:18:46.797 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: Created symlink /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715.target.wants/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg.service → /etc/systemd/system/ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@.service. 2024-01-18T07:18:46.880 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mgr.smithi084.sihbqg 2024-01-18T07:18:47.231 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T07:18:47.240 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-18T07:18:47.241 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T07:18:47.248 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:18:47.249 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-18T07:18:47.249 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to enable service . firewalld.service is not available 2024-01-18T07:18:47.249 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-18T07:18:47.250 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T07:18:47.257 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-18T07:18:47.258 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T07:18:47.265 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:18:47.266 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-18T07:18:47.267 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-18T07:18:47.267 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr to start... 2024-01-18T07:18:47.267 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr... 2024-01-18T07:18:47.267 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T07:18:47.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:46.303441+0000 mon.smithi084 (mon.0) 6 : audit [INF] from='client.? 172.21.15.84:0/1691222189' entity='client.admin' 2024-01-18T07:18:47.646 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.645+0000 7fd9e26b5700 1 Processor -- start 2024-01-18T07:18:47.647 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.645+0000 7fd9e26b5700 1 -- start start 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.645+0000 7fd9e26b5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.645+0000 7fd9e26b5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc0fb2e0 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9e16b3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9e16b3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52300/0 (socket says 172.21.15.84:52300) 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9e16b3700 1 -- 172.21.15.84:0/789815841 learned_addr learned my addr 172.21.15.84:0/789815841 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9e16b3700 1 -- 172.21.15.84:0/789815841 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc0fb420 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9e16b3700 1 --2- 172.21.15.84:0/789815841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd9d800b5f0 tx=0x7fd9d8006ee0).ready entity=mon.0 client_cookie=f726c339c71eec6b server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9cbfff700 1 -- 172.21.15.84:0/789815841 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d800d600 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9cbfff700 1 -- 172.21.15.84:0/789815841 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd9d800dbd0 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.648 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.646+0000 7fd9cbfff700 1 -- 172.21.15.84:0/789815841 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d8011c00 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.651 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.647+0000 7fd9e26b5700 1 -- 172.21.15.84:0/789815841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 msgr2=0x7fd9dc0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:47.651 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.647+0000 7fd9e26b5700 1 --2- 172.21.15.84:0/789815841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd9d800b5f0 tx=0x7fd9d8006ee0).stop 2024-01-18T07:18:47.651 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.648+0000 7fd9e26b5700 1 -- 172.21.15.84:0/789815841 shutdown_connections 2024-01-18T07:18:47.651 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.648+0000 7fd9e26b5700 1 --2- 172.21.15.84:0/789815841 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc0fadf0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:47.651 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.648+0000 7fd9e26b5700 1 -- 172.21.15.84:0/789815841 >> 172.21.15.84:0/789815841 conn(0x7fd9dc0f6210 msgr2=0x7fd9dc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.648+0000 7fd9e26b5700 1 -- 172.21.15.84:0/789815841 shutdown_connections 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.648+0000 7fd9e26b5700 1 -- 172.21.15.84:0/789815841 wait complete. 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e26b5700 1 Processor -- start 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e26b5700 1 -- start start 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e26b5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e26b5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9dc108b30 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e16b3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e16b3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52302/0 (socket says 172.21.15.84:52302) 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e16b3700 1 -- 172.21.15.84:0/1091751879 learned_addr learned my addr 172.21.15.84:0/1091751879 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e16b3700 1 -- 172.21.15.84:0/1091751879 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d800b200 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e16b3700 1 --2- 172.21.15.84:0/1091751879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd9d800e040 tx=0x7fd9d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:47.652 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d801b410 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd9d801b9e0 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9dc108d30 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.649+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d801a820 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.650+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9dc109140 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.650+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd9d800f070 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.650+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd9d8025ae0 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.651+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9dc045ad0 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.653 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.653+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd9d8016070 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.721+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd9dc102420 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.721+0000 7fd9c9ffb700 1 -- 172.21.15.84:0/1091751879 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd9d8006a70 con 0x7fd9dc0fa9d0 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "bf9448d2-b5d1-11ee-95ae-87774f69a715", 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T07:18:47.729 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T07:18:47.730 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.731 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T07:18:47.732 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T07:18:47.733 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-18T07:18:42.512019+0000", 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 msgr2=0x7fd9dc108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 --2- 172.21.15.84:0/1091751879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd9d800e040 tx=0x7fd9d8006ee0).stop 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 shutdown_connections 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 --2- 172.21.15.84:0/1091751879 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9dc0fa9d0 0x7fd9dc108640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 >> 172.21.15.84:0/1091751879 conn(0x7fd9dc0f6210 msgr2=0x7fd9dc0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 shutdown_connections 2024-01-18T07:18:47.734 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:47.723+0000 7fd9e26b5700 1 -- 172.21.15.84:0/1091751879 wait complete. 2024-01-18T07:18:48.128 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (1/15)... 2024-01-18T07:18:48.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:47.722488+0000 mon.smithi084 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.84:0/1091751879' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T07:18:50.131 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T07:18:50.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.558+0000 7fd8e283f700 1 Processor -- start 2024-01-18T07:18:50.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.558+0000 7fd8e283f700 1 -- start start 2024-01-18T07:18:50.560 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.558+0000 7fd8e283f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.558+0000 7fd8e283f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc0fb3b0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.558+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52314/0 (socket says 172.21.15.84:52314) 2024-01-18T07:18:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8dbfff700 1 -- 172.21.15.84:0/879098112 learned_addr learned my addr 172.21.15.84:0/879098112 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:50.561 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8dbfff700 1 -- 172.21.15.84:0/879098112 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc0fb4f0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8dbfff700 1 --2- 172.21.15.84:0/879098112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd8cc00b5f0 tx=0x7fd8cc006ee0).ready entity=mon.0 client_cookie=673023cb763d2d6a server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8daffd700 1 -- 172.21.15.84:0/879098112 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8cc00d600 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8daffd700 1 -- 172.21.15.84:0/879098112 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd8cc00dbd0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.559+0000 7fd8daffd700 1 -- 172.21.15.84:0/879098112 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8cc011c00 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.560+0000 7fd8e283f700 1 -- 172.21.15.84:0/879098112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 msgr2=0x7fd8dc0faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.560+0000 7fd8e283f700 1 --2- 172.21.15.84:0/879098112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd8cc00b5f0 tx=0x7fd8cc006ee0).stop 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 -- 172.21.15.84:0/879098112 shutdown_connections 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 --2- 172.21.15.84:0/879098112 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc0faec0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 -- 172.21.15.84:0/879098112 >> 172.21.15.84:0/879098112 conn(0x7fd8dc0f62a0 msgr2=0x7fd8dc0f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 -- 172.21.15.84:0/879098112 shutdown_connections 2024-01-18T07:18:50.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 -- 172.21.15.84:0/879098112 wait complete. 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.561+0000 7fd8e283f700 1 Processor -- start 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8e283f700 1 -- start start 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8e283f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8e283f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc10d070 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:50.563 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8dbfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52322/0 (socket says 172.21.15.84:52322) 2024-01-18T07:18:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8dbfff700 1 -- 172.21.15.84:0/769566898 learned_addr learned my addr 172.21.15.84:0/769566898 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8dbfff700 1 -- 172.21.15.84:0/769566898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8cc00b200 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.562+0000 7fd8dbfff700 1 --2- 172.21.15.84:0/769566898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd8cc00e040 tx=0x7fd8cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8cc01b410 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc10d270 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd8cc01b9e0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8cc01a770 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc10d700 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd8cc00f070 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.563+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd8cc024b50 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.564+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc045ad0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.566+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd8cc016070 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.634 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.633+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd8dc0fc810 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.633+0000 7fd8d8ff9700 1 -- 172.21.15.84:0/769566898 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd8cc024de0 con 0x7fd8dc0faaa0 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "bf9448d2-b5d1-11ee-95ae-87774f69a715", 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T07:18:50.635 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T07:18:50.642 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T07:18:50.643 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": false, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T07:18:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-18T07:18:42.512019+0000", 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 msgr2=0x7fd8dc10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 --2- 172.21.15.84:0/769566898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd8cc00e040 tx=0x7fd8cc006ee0).stop 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 shutdown_connections 2024-01-18T07:18:50.645 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 --2- 172.21.15.84:0/769566898 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd8dc0faaa0 0x7fd8dc10cb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:50.646 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 >> 172.21.15.84:0/769566898 conn(0x7fd8dc0f62a0 msgr2=0x7fd8dc070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:50.646 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 shutdown_connections 2024-01-18T07:18:50.646 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:50.634+0000 7fd8e283f700 1 -- 172.21.15.84:0/769566898 wait complete. 2024-01-18T07:18:50.930 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:50.634203+0000 mon.smithi084 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.84:0/769566898' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T07:18:50.986 INFO:teuthology.orchestra.run.smithi084.stderr:mgr not available, waiting (2/15)... 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:52.160745+0000 mon.smithi084 (mon.0) 9 : cluster [INF] Activating manager daemon smithi084.sihbqg 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:52.163449+0000 mon.smithi084 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi084.sihbqg(active, starting, since 0.00280856s) 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.164351+0000 mon.smithi084 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.164464+0000 mon.smithi084 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.164565+0000 mon.smithi084 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T07:18:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.164682+0000 mon.smithi084 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.165090+0000 mon.smithi084 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi084.sihbqg", "id": "smithi084.sihbqg"}]: dispatch 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:52.178088+0000 mon.smithi084 (mon.0) 16 : cluster [INF] Manager daemon smithi084.sihbqg is now available 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.184865+0000 mon.smithi084 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.189494+0000 mon.smithi084 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.190565+0000 mon.smithi084 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.195151+0000 mon.smithi084 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' 2024-01-18T07:18:52.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:52.197275+0000 mon.smithi084 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.84:0/3656976786' entity='mgr.smithi084.sihbqg' 2024-01-18T07:18:52.989 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.345+0000 7f6048d27700 1 Processor -- start 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.345+0000 7f6048d27700 1 -- start start 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f6048d27700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f6048d27700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440fe7f0 con 0x7f60440fdee0 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52386/0 (socket says 172.21.15.84:52386) 2024-01-18T07:18:53.349 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604259c700 1 -- 172.21.15.84:0/327797742 learned_addr learned my addr 172.21.15.84:0/327797742 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604259c700 1 -- 172.21.15.84:0/327797742 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440fe930 con 0x7f60440fdee0 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604259c700 1 --2- 172.21.15.84:0/327797742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f603400b5f0 tx=0x7f6034006ee0).ready entity=mon.0 client_cookie=a69390e329cec2ac server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604159a700 1 -- 172.21.15.84:0/327797742 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f603400d600 con 0x7f60440fdee0 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604159a700 1 -- 172.21.15.84:0/327797742 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f603400dbd0 con 0x7f60440fdee0 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.346+0000 7f604159a700 1 -- 172.21.15.84:0/327797742 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6034011c00 con 0x7f60440fdee0 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 -- 172.21.15.84:0/327797742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 msgr2=0x7f60440fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 --2- 172.21.15.84:0/327797742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f603400b5f0 tx=0x7f6034006ee0).stop 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 -- 172.21.15.84:0/327797742 shutdown_connections 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 --2- 172.21.15.84:0/327797742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f60440fe300 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 -- 172.21.15.84:0/327797742 >> 172.21.15.84:0/327797742 conn(0x7f60440714f0 msgr2=0x7f6044071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:53.350 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 -- 172.21.15.84:0/327797742 shutdown_connections 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.347+0000 7f6048d27700 1 -- 172.21.15.84:0/327797742 wait complete. 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6048d27700 1 Processor -- start 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6048d27700 1 -- start start 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6048d27700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6048d27700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d090 con 0x7f60440fdee0 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f604259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:53.351 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f604259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52390/0 (socket says 172.21.15.84:52390) 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f604259c700 1 -- 172.21.15.84:0/1461765845 learned_addr learned my addr 172.21.15.84:0/1461765845 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f604259c700 1 -- 172.21.15.84:0/1461765845 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603400b200 con 0x7f60440fdee0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f604259c700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f603400b1d0 tx=0x7f6034006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f603401b410 con 0x7f60440fdee0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604410d290 con 0x7f60440fdee0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f603401b9e0 con 0x7f60440fdee0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.348+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f603401a840 con 0x7f60440fdee0 2024-01-18T07:18:53.352 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.349+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604410d6a0 con 0x7f60440fdee0 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.349+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f603400f070 con 0x7f60440fdee0 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.349+0000 7f6032ffd700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 0x7f6028032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.349+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6034041160 con 0x7f60440fdee0 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.349+0000 7f6041d9b700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 0x7f6028032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.350+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6044059100 con 0x7f60440fdee0 2024-01-18T07:18:53.353 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.350+0000 7f6041d9b700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 0x7f6028032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f60380096a0 tx=0x7f6038006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:53.354 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.354+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6034016070 con 0x7f60440fdee0 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.555+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6044045ad0 con 0x7f60440fdee0 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.556+0000 7f6032ffd700 1 -- 172.21.15.84:0/1461765845 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f6034025460 con 0x7f60440fdee0 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsid": "bf9448d2-b5d1-11ee-95ae-87774f69a715", 2024-01-18T07:18:53.564 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "health": { 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "checks": {}, 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mutes": [] 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum": [ 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 0 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-18T07:18:53.565 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "smithi084" 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "monmap": { 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osdmap": { 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-18T07:18:53.566 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgmap": { 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "fsmap": { 2024-01-18T07:18:53.567 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modules": [ 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "iostat", 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "restful" 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ], 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.568 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "servicemap": { 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "modified": "2024-01-18T07:18:42.512019+0000", 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "services": {} 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: }, 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 msgr2=0x7f6028032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 0x7f6028032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f60380096a0 tx=0x7f6038006b40).stop 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 msgr2=0x7f604410cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f603400b1d0 tx=0x7f6034006ee0).stop 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 shutdown_connections 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7f602802fc50 0x7f6028032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 --2- 172.21.15.84:0/1461765845 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f60440fdee0 0x7f604410cba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 >> 172.21.15.84:0/1461765845 conn(0x7f60440714f0 msgr2=0x7f60440fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:53.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 shutdown_connections 2024-01-18T07:18:53.570 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:53.557+0000 7f6048d27700 1 -- 172.21.15.84:0/1461765845 wait complete. 2024-01-18T07:18:53.919 INFO:teuthology.orchestra.run.smithi084.stderr:mgr is available 2024-01-18T07:18:53.919 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling cephadm module... 2024-01-18T07:18:53.919 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-18T07:18:54.265 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.262+0000 7fe803593700 1 Processor -- start 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.262+0000 7fe803593700 1 -- start start 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.262+0000 7fe803593700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.262+0000 7fe803593700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0fee10 con 0x7fe7fc0703f0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.262+0000 7fe80132f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe80132f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52394/0 (socket says 172.21.15.84:52394) 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe80132f700 1 -- 172.21.15.84:0/588241123 learned_addr learned my addr 172.21.15.84:0/588241123 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe80132f700 1 -- 172.21.15.84:0/588241123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0fef50 con 0x7fe7fc0703f0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe80132f700 1 --2- 172.21.15.84:0/588241123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe7ec00b5f0 tx=0x7fe7ec006ee0).ready entity=mon.0 client_cookie=eda303339bc60d47 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe7ebfff700 1 -- 172.21.15.84:0/588241123 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec00d600 con 0x7fe7fc0703f0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe7ebfff700 1 -- 172.21.15.84:0/588241123 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7ec00dbd0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.263+0000 7fe7ebfff700 1 -- 172.21.15.84:0/588241123 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec011c00 con 0x7fe7fc0703f0 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- 172.21.15.84:0/588241123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 msgr2=0x7fe7fc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 --2- 172.21.15.84:0/588241123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe7ec00b5f0 tx=0x7fe7ec006ee0).stop 2024-01-18T07:18:54.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- 172.21.15.84:0/588241123 shutdown_connections 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 --2- 172.21.15.84:0/588241123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc0fe920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- 172.21.15.84:0/588241123 >> 172.21.15.84:0/588241123 conn(0x7fe7fc0f63c0 msgr2=0x7fe7fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- 172.21.15.84:0/588241123 shutdown_connections 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- 172.21.15.84:0/588241123 wait complete. 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 Processor -- start 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.264+0000 7fe803593700 1 -- start start 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe803593700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe803593700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc10aeb0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe80132f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe80132f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52406/0 (socket says 172.21.15.84:52406) 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe80132f700 1 -- 172.21.15.84:0/966182934 learned_addr learned my addr 172.21.15.84:0/966182934 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe80132f700 1 -- 172.21.15.84:0/966182934 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec00b200 con 0x7fe7fc0703f0 2024-01-18T07:18:54.267 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe80132f700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe7ec00b190 tx=0x7fe7ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:54.269 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec01b410 con 0x7fe7fc0703f0 2024-01-18T07:18:54.269 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7ec01b9e0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.269 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc10b0b0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.269 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec01a880 con 0x7fe7fc0703f0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.265+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc10b4c0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.266+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fe7ec00f070 con 0x7fe7fc0703f0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.266+0000 7fe7e9ffb700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 0x7fe7e4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.266+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe7ec0410d0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.266+0000 7fe800b2e700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 0x7fe7e4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.267+0000 7fe800b2e700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 0x7fe7e4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe7f80096a0 tx=0x7fe7f8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:54.270 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.267+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc059100 con 0x7fe7fc0703f0 2024-01-18T07:18:54.271 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.271+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7ec016070 con 0x7fe7fc0703f0 2024-01-18T07:18:54.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:54.455+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe7fc045ad0 con 0x7fe7fc0703f0 2024-01-18T07:18:54.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:53.166350+0000 mon.smithi084 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi084.sihbqg(active, since 1.00571s) 2024-01-18T07:18:54.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:53.556911+0000 mon.smithi084 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.84:0/1461765845' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-18T07:18:55.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.171+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe7ec0257e0 con 0x7fe7fc0703f0 2024-01-18T07:18:55.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.172+0000 7fe7e9ffb700 1 -- 172.21.15.84:0/966182934 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe7ec025460 con 0x7fe7fc0703f0 2024-01-18T07:18:55.175 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 msgr2=0x7fe7e4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:55.176 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 0x7fe7e4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe7f80096a0 tx=0x7fe7f8006b40).stop 2024-01-18T07:18:55.176 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 msgr2=0x7fe7fc10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:55.176 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fe7ec00b190 tx=0x7fe7ec006ee0).stop 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 shutdown_connections 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fe7e402fd50 0x7fe7e4032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 --2- 172.21.15.84:0/966182934 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe7fc0703f0 0x7fe7fc10a9c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.175+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 >> 172.21.15.84:0/966182934 conn(0x7fe7fc0f63c0 msgr2=0x7fe7fc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.176+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 shutdown_connections 2024-01-18T07:18:55.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.176+0000 7fe803593700 1 -- 172.21.15.84:0/966182934 wait complete. 2024-01-18T07:18:55.512 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:54.169505+0000 mon.smithi084 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi084.sihbqg(active, since 2s) 2024-01-18T07:18:55.512 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:54.456043+0000 mon.smithi084 (mon.0) 25 : audit [INF] from='client.? 172.21.15.84:0/966182934' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-18T07:18:55.543 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.914+0000 7fee16737700 1 Processor -- start 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee16737700 1 -- start start 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee16737700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee16737700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee100fb490 con 0x7fee100fab80 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52440/0 (socket says 172.21.15.84:52440) 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0ffff700 1 -- 172.21.15.84:0/825878521 learned_addr learned my addr 172.21.15.84:0/825878521 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0ffff700 1 -- 172.21.15.84:0/825878521 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100fb5d0 con 0x7fee100fab80 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0ffff700 1 --2- 172.21.15.84:0/825878521 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fee0000b5f0 tx=0x7fee00006ee0).ready entity=mon.0 client_cookie=ade39cb3bcfde5f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:55.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.915+0000 7fee0effd700 1 -- 172.21.15.84:0/825878521 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee0000d600 con 0x7fee100fab80 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee0effd700 1 -- 172.21.15.84:0/825878521 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fee0000dbd0 con 0x7fee100fab80 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee0effd700 1 -- 172.21.15.84:0/825878521 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee00011c00 con 0x7fee100fab80 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 -- 172.21.15.84:0/825878521 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 msgr2=0x7fee100fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 --2- 172.21.15.84:0/825878521 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fee0000b5f0 tx=0x7fee00006ee0).stop 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 -- 172.21.15.84:0/825878521 shutdown_connections 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 --2- 172.21.15.84:0/825878521 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee100fafa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 -- 172.21.15.84:0/825878521 >> 172.21.15.84:0/825878521 conn(0x7fee100f63c0 msgr2=0x7fee100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 -- 172.21.15.84:0/825878521 shutdown_connections 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.916+0000 7fee16737700 1 -- 172.21.15.84:0/825878521 wait complete. 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee16737700 1 Processor -- start 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee16737700 1 -- start start 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee16737700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee16737700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1010d1b0 con 0x7fee100fab80 2024-01-18T07:18:55.920 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:52456/0 (socket says 172.21.15.84:52456) 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.917+0000 7fee0ffff700 1 -- 172.21.15.84:0/1476065479 learned_addr learned my addr 172.21.15.84:0/1476065479 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee0ffff700 1 -- 172.21.15.84:0/1476065479 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0000b200 con 0x7fee100fab80 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee0ffff700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fee0000e040 tx=0x7fee00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee0001b410 con 0x7fee100fab80 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fee0001b9e0 con 0x7fee100fab80 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1010d3b0 con 0x7fee100fab80 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee0001a820 con 0x7fee100fab80 2024-01-18T07:18:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.918+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1010d7c0 con 0x7fee100fab80 2024-01-18T07:18:55.922 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10045ad0 con 0x7fee100fab80 2024-01-18T07:18:55.922 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fee0000f070 con 0x7fee100fab80 2024-01-18T07:18:55.922 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee0cff9700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 0x7fedfc032200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:55.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fee00041320 con 0x7fee100fab80 2024-01-18T07:18:55.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee0f7fe700 1 -- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 msgr2=0x7fedfc032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:55.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.919+0000 7fee0f7fe700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 0x7fedfc032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:18:55.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:55.923+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee00022b90 con 0x7fee100fab80 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.114+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fee1010e660 con 0x7fee100fab80 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.115+0000 7fee0cff9700 1 -- 172.21.15.84:0/1476065479 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fee00016070 con 0x7fee100fab80 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-18T07:18:56.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.sihbqg", 2024-01-18T07:18:56.117 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-18T07:18:56.117 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 msgr2=0x7fedfc032200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 0x7fedfc032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 msgr2=0x7fee1010ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fee0000e040 tx=0x7fee00006ee0).stop 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 shutdown_connections 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fedfc02fd40 0x7fedfc032200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 --2- 172.21.15.84:0/1476065479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fee100fab80 0x7fee1010ccc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:56.118 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 >> 172.21.15.84:0/1476065479 conn(0x7fee100f63c0 msgr2=0x7fee100ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:56.119 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 shutdown_connections 2024-01-18T07:18:56.119 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.117+0000 7fee16737700 1 -- 172.21.15.84:0/1476065479 wait complete. 2024-01-18T07:18:56.447 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:55.173230+0000 mon.smithi084 (mon.0) 26 : audit [INF] from='client.? 172.21.15.84:0/966182934' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-18T07:18:56.447 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:18:55.173288+0000 mon.smithi084 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi084.sihbqg(active, since 3s) 2024-01-18T07:18:56.447 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:18:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:18:56.115985+0000 mon.smithi084 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.84:0/1476065479' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-18T07:18:56.462 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-01-18T07:18:56.462 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 5... 2024-01-18T07:18:56.462 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-18T07:18:56.860 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.857+0000 7fa5d3c5e700 1 Processor -- start 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.857+0000 7fa5d3c5e700 1 -- start start 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d3c5e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d3c5e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0fb380 con 0x7fa5cc0faa70 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d19fa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d19fa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59530/0 (socket says 172.21.15.84:59530) 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d19fa700 1 -- 172.21.15.84:0/2868464605 learned_addr learned my addr 172.21.15.84:0/2868464605 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d19fa700 1 -- 172.21.15.84:0/2868464605 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0fb4c0 con 0x7fa5cc0faa70 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d19fa700 1 --2- 172.21.15.84:0/2868464605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa5c800b5f0 tx=0x7fa5c8006ee0).ready entity=mon.0 client_cookie=8d07b4ec35d16c6c server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d09f8700 1 -- 172.21.15.84:0/2868464605 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c800d600 con 0x7fa5cc0faa70 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d09f8700 1 -- 172.21.15.84:0/2868464605 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5c800dbd0 con 0x7fa5cc0faa70 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.858+0000 7fa5d09f8700 1 -- 172.21.15.84:0/2868464605 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c8011c00 con 0x7fa5cc0faa70 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/2868464605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 msgr2=0x7fa5cc0fae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:18:56.861 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/2868464605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fa5c800b5f0 tx=0x7fa5c8006ee0).stop 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/2868464605 shutdown_connections 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/2868464605 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc0fae90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/2868464605 >> 172.21.15.84:0/2868464605 conn(0x7fa5cc0f6290 msgr2=0x7fa5cc0f86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/2868464605 shutdown_connections 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.859+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/2868464605 wait complete. 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.860+0000 7fa5d3c5e700 1 Processor -- start 2024-01-18T07:18:56.862 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.860+0000 7fa5d3c5e700 1 -- start start 2024-01-18T07:18:56.864 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.860+0000 7fa5d3c5e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:56.864 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.860+0000 7fa5d3c5e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc06fc30 con 0x7fa5cc0faa70 2024-01-18T07:18:56.864 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d19fa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:18:56.864 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d19fa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59534/0 (socket says 172.21.15.84:59534) 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d19fa700 1 -- 172.21.15.84:0/1668243617 learned_addr learned my addr 172.21.15.84:0/1668243617 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d19fa700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c800b200 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d19fa700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa5c800e040 tx=0x7fa5c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c801b410 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5c801b9e0 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc06fe30 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5c801a820 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.861+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc10dd30 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa5c800f070 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5ba7fc700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5b40328d0 con 0x7fa5b402fd50 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa5c80412f0 con 0x7fa5cc0faa70 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5d11f9700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:56.865 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:56.862+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:18:57.064 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:57.063+0000 7fa5d11f9700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:57.064 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:57.063+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T07:18:57.464 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:57.464+0000 7fa5d11f9700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:57.465 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:57.464+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T07:18:58.265 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:58.265+0000 7fa5d11f9700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:58.266 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:58.265+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T07:18:59.866 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:59.866+0000 7fa5d11f9700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2132670921 2024-01-18T07:18:59.867 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:18:59.866+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T07:19:00.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:00.169+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7fa5c801aa50 con 0x7fa5cc0faa70 2024-01-18T07:19:00.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:00.169+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 msgr2=0x7fa5b4032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:19:00.170 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:00.169+0000 7fa5ba7fc700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:01.172 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.171+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fa5c8042bc0 con 0x7fa5cc0faa70 2024-01-18T07:19:01.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.172+0000 7fa5ba7fc700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 0x7fa5b4035c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:01.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.172+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5b40328d0 con 0x7fa5b4033820 2024-01-18T07:19:01.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.172+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 0x7fa5b4035c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:01.173 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.173+0000 7fa5d11f9700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 0x7fa5b4035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa5bc006fd0 tx=0x7fa5bc006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:01.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.173+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa5b40328d0 con 0x7fa5b4033820 2024-01-18T07:19:01.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.179+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa5cc059100 con 0x7fa5b4033820 2024-01-18T07:19:01.180 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5ba7fc700 1 -- 172.21.15.84:0/1668243617 <== mgr.14114 v2:172.21.15.84:6800/2693695158 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa5cc059100 con 0x7fa5b4033820 2024-01-18T07:19:01.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:19:01.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-18T07:19:01.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-01-18T07:19:01.181 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:19:01.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 msgr2=0x7fa5b4035c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:01.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 0x7fa5b4035c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa5bc006fd0 tx=0x7fa5bc006cf0).stop 2024-01-18T07:19:01.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 msgr2=0x7fa5cc10d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:01.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa5c800e040 tx=0x7fa5c8006ee0).stop 2024-01-18T07:19:01.182 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 shutdown_connections 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5cc0faa70 0x7fa5cc10d680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5b4033820 0x7fa5b4035c10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 --2- 172.21.15.84:0/1668243617 >> [v2:172.21.15.84:6800/2132670921,v1:172.21.15.84:6801/2132670921] conn(0x7fa5b402fd50 0x7fa5b4032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 >> 172.21.15.84:0/1668243617 conn(0x7fa5cc0f6290 msgr2=0x7fa5cc0f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 shutdown_connections 2024-01-18T07:19:01.183 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:01.180+0000 7fa5d3c5e700 1 -- 172.21.15.84:0/1668243617 wait complete. 2024-01-18T07:19:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:00.115707+0000 mon.smithi084 (mon.0) 29 : cluster [INF] Active manager daemon smithi084.sihbqg restarted 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:00.116201+0000 mon.smithi084 (mon.0) 30 : cluster [INF] Activating manager daemon smithi084.sihbqg 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:00.118060+0000 mon.smithi084 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:00.171419+0000 mon.smithi084 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi084.sihbqg(active, starting, since 0.0553304s) 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.171972+0000 mon.smithi084 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.172550+0000 mon.smithi084 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi084.sihbqg", "id": "smithi084.sihbqg"}]: dispatch 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.173148+0000 mon.smithi084 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.173300+0000 mon.smithi084 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.173413+0000 mon.smithi084 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:00.187048+0000 mon.smithi084 (mon.0) 38 : cluster [INF] Manager daemon smithi084.sihbqg is now available 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.200569+0000 mon.smithi084 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:01.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.205938+0000 mon.smithi084 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.208402+0000 mon.smithi084 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.295757+0000 mon.smithi084 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.296432+0000 mon.smithi084 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.297523+0000 mon.smithi084 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.299314+0000 mon.smithi084 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.309086+0000 mon.smithi084 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:19:01.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:00.310191+0000 mon.smithi084 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:19:01.784 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 5 is available 2024-01-18T07:19:01.784 INFO:teuthology.orchestra.run.smithi084.stderr:Setting orchestrator backend to cephadm... 2024-01-18T07:19:01.784 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-18T07:19:02.134 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.130+0000 7f8fc9d43700 1 Processor -- start 2024-01-18T07:19:02.135 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.130+0000 7f8fc9d43700 1 -- start start 2024-01-18T07:19:02.135 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc9d43700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:02.135 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc9d43700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc40fb490 con 0x7f8fc40fab80 2024-01-18T07:19:02.135 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:02.135 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59592/0 (socket says 172.21.15.84:59592) 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc37fe700 1 -- 172.21.15.84:0/486993655 learned_addr learned my addr 172.21.15.84:0/486993655 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc37fe700 1 -- 172.21.15.84:0/486993655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc40fb5d0 con 0x7f8fc40fab80 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc37fe700 1 --2- 172.21.15.84:0/486993655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8fb800b870 tx=0x7f8fb8006f70).ready entity=mon.0 client_cookie=7a5980d2aaccfbbd server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc27fc700 1 -- 172.21.15.84:0/486993655 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb800da20 con 0x7f8fc40fab80 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc27fc700 1 -- 172.21.15.84:0/486993655 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8fb8010470 con 0x7f8fc40fab80 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.131+0000 7f8fc27fc700 1 -- 172.21.15.84:0/486993655 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb800f5b0 con 0x7f8fc40fab80 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 -- 172.21.15.84:0/486993655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 msgr2=0x7f8fc40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/486993655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f8fb800b870 tx=0x7f8fb8006f70).stop 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 -- 172.21.15.84:0/486993655 shutdown_connections 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/486993655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc40fafa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:02.136 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 -- 172.21.15.84:0/486993655 >> 172.21.15.84:0/486993655 conn(0x7f8fc40f63c0 msgr2=0x7f8fc40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 -- 172.21.15.84:0/486993655 shutdown_connections 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 -- 172.21.15.84:0/486993655 wait complete. 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.132+0000 7f8fc9d43700 1 Processor -- start 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc9d43700 1 -- start start 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc9d43700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc9d43700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc410d080 con 0x7f8fc40fab80 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59594/0 (socket says 172.21.15.84:59594) 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc37fe700 1 -- 172.21.15.84:0/1898230927 learned_addr learned my addr 172.21.15.84:0/1898230927 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc37fe700 1 -- 172.21.15.84:0/1898230927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb800b480 con 0x7f8fc40fab80 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fc37fe700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8fb8006a40 tx=0x7f8fb8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.133+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb800de00 con 0x7f8fc40fab80 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc410d280 con 0x7f8fc40fab80 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8fb800fef0 con 0x7f8fc40fab80 2024-01-18T07:19:02.137 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fb8019e90 con 0x7f8fc40fab80 2024-01-18T07:19:02.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc410d690 con 0x7f8fc40fab80 2024-01-18T07:19:02.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f8fb802a070 con 0x7f8fc40fab80 2024-01-18T07:19:02.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.134+0000 7f8fabfff700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 0x7f8fb0032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:02.138 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.135+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc4045ad0 con 0x7f8fc40fab80 2024-01-18T07:19:02.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.135+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8fb80419d0 con 0x7f8fc40fab80 2024-01-18T07:19:02.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.135+0000 7f8fc2ffd700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 0x7f8fb0032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:02.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.135+0000 7f8fc2ffd700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 0x7f8fb0032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8fb40096a0 tx=0x7f8fb4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:02.139 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.139+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fb8014020 con 0x7f8fc40fab80 2024-01-18T07:19:02.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.174+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8fb8016030 con 0x7f8fc40fab80 2024-01-18T07:19:02.318 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.317+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fc410dfd0 con 0x7f8fb002fca0 2024-01-18T07:19:02.323 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.322+0000 7f8fabfff700 1 -- 172.21.15.84:0/1898230927 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8fc410dfd0 con 0x7f8fb002fca0 2024-01-18T07:19:02.325 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 msgr2=0x7f8fb0032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:02.325 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 0x7f8fb0032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8fb40096a0 tx=0x7f8fb4006b40).stop 2024-01-18T07:19:02.325 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 msgr2=0x7f8fc410cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8fb8006a40 tx=0x7f8fb8006f70).stop 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 shutdown_connections 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8fb002fca0 0x7f8fb0032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 --2- 172.21.15.84:0/1898230927 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8fc40fab80 0x7f8fc410cb90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.324+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 >> 172.21.15.84:0/1898230927 conn(0x7f8fc40f63c0 msgr2=0x7f8fc40ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.325+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 shutdown_connections 2024-01-18T07:19:02.326 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:02.325+0000 7f8fc9d43700 1 -- 172.21.15.84:0/1898230927 wait complete. 2024-01-18T07:19:02.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:01.174223+0000 mgr.smithi084.sihbqg (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-18T07:19:02.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:01.174937+0000 mon.smithi084 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi084.sihbqg(active, since 1.05885s) 2024-01-18T07:19:02.549 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:01.180915+0000 mgr.smithi084.sihbqg (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-18T07:19:02.701 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-18T07:19:03.050 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.045+0000 7f3d75c39700 1 Processor -- start 2024-01-18T07:19:03.050 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.045+0000 7f3d75c39700 1 -- start start 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.045+0000 7f3d75c39700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.045+0000 7f3d75c39700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700fcb80 con 0x7f3d700fc270 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59596/0 (socket says 172.21.15.84:59596) 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6f7fe700 1 -- 172.21.15.84:0/984840131 learned_addr learned my addr 172.21.15.84:0/984840131 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6f7fe700 1 -- 172.21.15.84:0/984840131 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700fccc0 con 0x7f3d700fc270 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6f7fe700 1 --2- 172.21.15.84:0/984840131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3d6400b5f0 tx=0x7f3d64006ee0).ready entity=mon.0 client_cookie=6c1435af95520a56 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6e7fc700 1 -- 172.21.15.84:0/984840131 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6400d600 con 0x7f3d700fc270 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6e7fc700 1 -- 172.21.15.84:0/984840131 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3d6400dbd0 con 0x7f3d700fc270 2024-01-18T07:19:03.051 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.046+0000 7f3d6e7fc700 1 -- 172.21.15.84:0/984840131 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d64011c00 con 0x7f3d700fc270 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 -- 172.21.15.84:0/984840131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 msgr2=0x7f3d700fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 --2- 172.21.15.84:0/984840131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3d6400b5f0 tx=0x7f3d64006ee0).stop 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 -- 172.21.15.84:0/984840131 shutdown_connections 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 --2- 172.21.15.84:0/984840131 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d700fc690 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 -- 172.21.15.84:0/984840131 >> 172.21.15.84:0/984840131 conn(0x7f3d700f5b40 msgr2=0x7f3d700f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 -- 172.21.15.84:0/984840131 shutdown_connections 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 -- 172.21.15.84:0/984840131 wait complete. 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.047+0000 7f3d75c39700 1 Processor -- start 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d75c39700 1 -- start start 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d75c39700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d75c39700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7010d1f0 con 0x7f3d700fc270 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d6f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59608/0 (socket says 172.21.15.84:59608) 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d6f7fe700 1 -- 172.21.15.84:0/3638708830 learned_addr learned my addr 172.21.15.84:0/3638708830 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d6f7fe700 1 -- 172.21.15.84:0/3638708830 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6400b200 con 0x7f3d700fc270 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.048+0000 7f3d6f7fe700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3d64006c70 tx=0x7f3d64006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6400d9f0 con 0x7f3d700fc270 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7010d3f0 con 0x7f3d700fc270 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3d6401a9f0 con 0x7f3d700fc270 2024-01-18T07:19:03.053 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6401b890 con 0x7f3d700fc270 2024-01-18T07:19:03.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7010d800 con 0x7f3d700fc270 2024-01-18T07:19:03.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.049+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3d6400f070 con 0x7f3d700fc270 2024-01-18T07:19:03.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.050+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d70045ad0 con 0x7f3d700fc270 2024-01-18T07:19:03.054 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.050+0000 7f3d57fff700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 0x7f3d5c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.055 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.050+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3d64041270 con 0x7f3d700fc270 2024-01-18T07:19:03.055 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.050+0000 7f3d6effd700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 0x7f3d5c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.055 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.050+0000 7f3d6effd700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 0x7f3d5c032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3d600096a0 tx=0x7f3d60006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.055 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.054+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d64022900 con 0x7f3d700fc270 2024-01-18T07:19:03.207 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.206+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d7010e060 con 0x7f3d5c02fd50 2024-01-18T07:19:03.208 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.207+0000 7f3d57fff700 1 -- 172.21.15.84:0/3638708830 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3d7010e060 con 0x7f3d5c02fd50 2024-01-18T07:19:03.208 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: value unchanged 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 msgr2=0x7f3d5c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 0x7f3d5c032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3d600096a0 tx=0x7f3d60006b40).stop 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 msgr2=0x7f3d7010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3d64006c70 tx=0x7f3d64006ee0).stop 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 shutdown_connections 2024-01-18T07:19:03.211 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f3d5c02fd50 0x7f3d5c032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:03.212 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 --2- 172.21.15.84:0/3638708830 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3d700fc270 0x7f3d7010cd00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:03.212 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 >> 172.21.15.84:0/3638708830 conn(0x7f3d700f5b40 msgr2=0x7f3d700f76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:03.212 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 shutdown_connections 2024-01-18T07:19:03.212 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.208+0000 7f3d75c39700 1 -- 172.21.15.84:0/3638708830 wait complete. 2024-01-18T07:19:03.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:02.176008+0000 mon.smithi084 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi084.sihbqg(active, since 2s) 2024-01-18T07:19:03.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:02.322892+0000 mon.smithi084 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:03.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:02.326152+0000 mon.smithi084 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:03.600 INFO:teuthology.orchestra.run.smithi084.stderr:Generating ssh key... 2024-01-18T07:19:03.601 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-18T07:19:03.971 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5fa2ae700 1 Processor -- start 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5fa2ae700 1 -- start start 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5fa2ae700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5fa2ae700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40fb490 con 0x7fa5f40fab80 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5f3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5f3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59616/0 (socket says 172.21.15.84:59616) 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5f3fff700 1 -- 172.21.15.84:0/1826367500 learned_addr learned my addr 172.21.15.84:0/1826367500 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5f3fff700 1 -- 172.21.15.84:0/1826367500 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40fb5d0 con 0x7fa5f40fab80 2024-01-18T07:19:03.975 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.970+0000 7fa5f3fff700 1 --2- 172.21.15.84:0/1826367500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa5e400b5f0 tx=0x7fa5e4006ee0).ready entity=mon.0 client_cookie=3c11ba4a6f74b9f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5f2ffd700 1 -- 172.21.15.84:0/1826367500 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e400d600 con 0x7fa5f40fab80 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5f2ffd700 1 -- 172.21.15.84:0/1826367500 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5e400dbd0 con 0x7fa5f40fab80 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5f2ffd700 1 -- 172.21.15.84:0/1826367500 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e4011c00 con 0x7fa5f40fab80 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/1826367500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 msgr2=0x7fa5f40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/1826367500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa5e400b5f0 tx=0x7fa5e4006ee0).stop 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/1826367500 shutdown_connections 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/1826367500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f40fafa0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/1826367500 >> 172.21.15.84:0/1826367500 conn(0x7fa5f40f63c0 msgr2=0x7fa5f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/1826367500 shutdown_connections 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.971+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/1826367500 wait complete. 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5fa2ae700 1 Processor -- start 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5fa2ae700 1 -- start start 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5fa2ae700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5fa2ae700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f410b510 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5f3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5f3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59620/0 (socket says 172.21.15.84:59620) 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5f3fff700 1 -- 172.21.15.84:0/3710817664 learned_addr learned my addr 172.21.15.84:0/3710817664 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.972+0000 7fa5f3fff700 1 -- 172.21.15.84:0/3710817664 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e400b200 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5f3fff700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa5e400b1d0 tx=0x7fa5e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e401b410 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f4070450 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5e401b9e0 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5e401a840 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4110010 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.973+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa5e400f070 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.974+0000 7fa5f0ff9700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 0x7fa5e0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.974+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa5e40412a0 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.974+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f4045ad0 con 0x7fa5f40fab80 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.974+0000 7fa5f37fe700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 0x7fa5e0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:03.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.974+0000 7fa5f37fe700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 0x7fa5e0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa5e80096a0 tx=0x7fa5e8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:03.978 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:03.978+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5e4025460 con 0x7fa5f40fab80 2024-01-18T07:19:04.128 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.128+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5f40fc900 con 0x7fa5e002fd50 2024-01-18T07:19:04.174 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.174+0000 7fa5f0ff9700 1 -- 172.21.15.84:0/3710817664 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa5f40fc900 con 0x7fa5e002fd50 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 msgr2=0x7fa5e0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 0x7fa5e0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa5e80096a0 tx=0x7fa5e8006b40).stop 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 msgr2=0x7fa5f410fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa5e400b1d0 tx=0x7fa5e4006ee0).stop 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 shutdown_connections 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa5e002fd50 0x7fa5e0032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 --2- 172.21.15.84:0/3710817664 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa5f40fab80 0x7fa5f410fb20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 >> 172.21.15.84:0/3710817664 conn(0x7fa5f40f63c0 msgr2=0x7fa5f40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:04.177 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 shutdown_connections 2024-01-18T07:19:04.178 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.176+0000 7fa5fa2ae700 1 -- 172.21.15.84:0/3710817664 wait complete. 2024-01-18T07:19:04.460 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:02.319153+0000 mgr.smithi084.sihbqg (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:04.460 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:03.207704+0000 mgr.smithi084.sihbqg (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:04.461 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:04.172409+0000 mon.smithi084 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:04.461 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:04.174140+0000 mon.smithi084 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:04.549 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.909+0000 7fa4b1f66700 1 Processor -- start 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.909+0000 7fa4b1f66700 1 -- start start 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.909+0000 7fa4b1f66700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.909+0000 7fa4b1f66700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0fc750 con 0x7fa4ac0703b0 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59626/0 (socket says 172.21.15.84:59626) 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4ab7fe700 1 -- 172.21.15.84:0/15454336 learned_addr learned my addr 172.21.15.84:0/15454336 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:04.914 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4ab7fe700 1 -- 172.21.15.84:0/15454336 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac101250 con 0x7fa4ac0703b0 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4ab7fe700 1 --2- 172.21.15.84:0/15454336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa49400b5f0 tx=0x7fa494006ee0).ready entity=mon.0 client_cookie=b2b8e2ebc59320bc server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4aa7fc700 1 -- 172.21.15.84:0/15454336 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49400d600 con 0x7fa4ac0703b0 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4aa7fc700 1 -- 172.21.15.84:0/15454336 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa49400dbd0 con 0x7fa4ac0703b0 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.910+0000 7fa4aa7fc700 1 -- 172.21.15.84:0/15454336 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa494011c00 con 0x7fa4ac0703b0 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- 172.21.15.84:0/15454336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 msgr2=0x7fa4ac100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/15454336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa49400b5f0 tx=0x7fa494006ee0).stop 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- 172.21.15.84:0/15454336 shutdown_connections 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/15454336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac100d60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- 172.21.15.84:0/15454336 >> 172.21.15.84:0/15454336 conn(0x7fa4ac0f63c0 msgr2=0x7fa4ac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- 172.21.15.84:0/15454336 shutdown_connections 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- 172.21.15.84:0/15454336 wait complete. 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 Processor -- start 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.911+0000 7fa4b1f66700 1 -- start start 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4b1f66700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:04.915 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4b1f66700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac10d100 con 0x7fa4ac0703b0 2024-01-18T07:19:04.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:04.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4ab7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59638/0 (socket says 172.21.15.84:59638) 2024-01-18T07:19:04.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4ab7fe700 1 -- 172.21.15.84:0/1620413192 learned_addr learned my addr 172.21.15.84:0/1620413192 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:04.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4ab7fe700 1 -- 172.21.15.84:0/1620413192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49400b200 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4ab7fe700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa49400b190 tx=0x7fa494006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49401b410 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa49401b9e0 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac10d300 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa49401a880 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.912+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac10d710 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.913+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa49400f070 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.913+0000 7fa4a3fff700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 0x7fa48c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.913+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa4940411b0 con 0x7fa4ac0703b0 2024-01-18T07:19:04.917 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.913+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac045ad0 con 0x7fa4ac0703b0 2024-01-18T07:19:04.918 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.914+0000 7fa4aaffd700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 0x7fa48c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:04.918 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.914+0000 7fa4aaffd700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 0x7fa48c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa49c0096a0 tx=0x7fa49c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:04.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:04.918+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa494025460 con 0x7fa4ac0703b0 2024-01-18T07:19:05.069 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.068+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4ac106b10 con 0x7fa48c02fdb0 2024-01-18T07:19:05.070 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.068+0000 7fa4a3fff700 1 -- 172.21.15.84:0/1620413192 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa4ac106b10 con 0x7fa48c02fdb0 2024-01-18T07:19:05.070 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0boFXm5J0Ld8wymGwzzh3OYqzU3nfzvrXmICElPGbhsqXOVZ6sSd05Upl+viYbj5HjIQPJjwzXzLLDtn4Zw4sMGnKKl2fjrX6f5CuWLhHKTYebLF0+DkeobS9p/YcGiVcpC4Et8DH0BsBWIPdsbbgWLOvaQORvdNVmlbjC5+Jn8GoVwxMzLJ5UE5pza7eHP6HqMRkYfYCoo7QVuqEQ2MPm9SVpW3cyaKOL/+u49a0JJed/dS7qhgmSJcAbnjLne/q8V4C3CzJcXmgGYXFbjcgoP2Df66HS7SNNkCpPy2uumXyGZdAvA5HalTSQCn02hOvyTINcF+QJTiQO3SRovXHHU7nX4zgeqMh2kqTZeUFhS2wgRgo1n42w23ViDW0UlnEylw1zomtPKSYIL4jah7lmhVnuxNLRTEYZ8fo5mPVFumYfWSHR6dcNDAEvGHRN4avT9RO5Eq/xmW+rSRxAa0EJuio69R88Rwe7PrcBrXbRS+URRKwrn3u8vDJIL14Up0= ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:19:05.072 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 msgr2=0x7fa48c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 0x7fa48c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa49c0096a0 tx=0x7fa49c006b40).stop 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 msgr2=0x7fa4ac10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa49400b190 tx=0x7fa494006ee0).stop 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 shutdown_connections 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fa48c02fdb0 0x7fa48c032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 --2- 172.21.15.84:0/1620413192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4ac0703b0 0x7fa4ac10cc10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 >> 172.21.15.84:0/1620413192 conn(0x7fa4ac0f63c0 msgr2=0x7fa4ac0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 shutdown_connections 2024-01-18T07:19:05.073 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.070+0000 7fa4b1f66700 1 -- 172.21.15.84:0/1620413192 wait complete. 2024-01-18T07:19:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:04.129369+0000 mgr.smithi084.sihbqg (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-18T07:19:05.458 INFO:teuthology.orchestra.run.smithi084.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-18T07:19:05.459 INFO:teuthology.orchestra.run.smithi084.stderr:Adding key to root@localhost authorized_keys... 2024-01-18T07:19:05.459 INFO:teuthology.orchestra.run.smithi084.stderr:Adding host smithi084... 2024-01-18T07:19:05.459 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi084 172.21.15.84 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.809+0000 7f4ef5d4c700 1 Processor -- start 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.809+0000 7f4ef5d4c700 1 -- start start 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.809+0000 7f4ef5d4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.809+0000 7f4ef5d4c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef00fc750 con 0x7f4ef00703b0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eef7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eef7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59644/0 (socket says 172.21.15.84:59644) 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eef7fe700 1 -- 172.21.15.84:0/1850571991 learned_addr learned my addr 172.21.15.84:0/1850571991 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eef7fe700 1 -- 172.21.15.84:0/1850571991 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef0101250 con 0x7f4ef00703b0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eef7fe700 1 --2- 172.21.15.84:0/1850571991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4ee000b5f0 tx=0x7f4ee0006ee0).ready entity=mon.0 client_cookie=9bc90e3c6a697e54 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eee7fc700 1 -- 172.21.15.84:0/1850571991 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee000d600 con 0x7f4ef00703b0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eee7fc700 1 -- 172.21.15.84:0/1850571991 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ee000dbd0 con 0x7f4ef00703b0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.810+0000 7f4eee7fc700 1 -- 172.21.15.84:0/1850571991 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee0011c00 con 0x7f4ef00703b0 2024-01-18T07:19:05.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/1850571991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 msgr2=0x7f4ef0100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/1850571991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4ee000b5f0 tx=0x7f4ee0006ee0).stop 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/1850571991 shutdown_connections 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/1850571991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef0100d60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/1850571991 >> 172.21.15.84:0/1850571991 conn(0x7f4ef00f63c0 msgr2=0x7f4ef00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/1850571991 shutdown_connections 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/1850571991 wait complete. 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.811+0000 7f4ef5d4c700 1 Processor -- start 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ef5d4c700 1 -- start start 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ef5d4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ef5d4c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef010d1e0 con 0x7f4ef00703b0 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4eef7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4eef7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59656/0 (socket says 172.21.15.84:59656) 2024-01-18T07:19:05.816 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4eef7fe700 1 -- 172.21.15.84:0/2761447338 learned_addr learned my addr 172.21.15.84:0/2761447338 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4eef7fe700 1 -- 172.21.15.84:0/2761447338 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee000b200 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4eef7fe700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4ee000e040 tx=0x7f4ee0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee000d9f0 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4ee001a9f0 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef010d3e0 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee001b8c0 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.812+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef010d7f0 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.813+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4ee000f070 con 0x7f4ef00703b0 2024-01-18T07:19:05.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.813+0000 7f4ed7fff700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 0x7f4edc032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:05.818 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.813+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4ee0042180 con 0x7f4ef00703b0 2024-01-18T07:19:05.818 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.813+0000 7f4eeeffd700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 0x7f4edc032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:05.818 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.814+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef0101db0 con 0x7f4ef00703b0 2024-01-18T07:19:05.818 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.814+0000 7f4eeeffd700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 0x7f4edc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4ee40096a0 tx=0x7f4ee4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:05.819 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.818+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ee0016070 con 0x7f4ef00703b0 2024-01-18T07:19:05.990 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:05.989+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ef0101f70 con 0x7f4edc02fdb0 2024-01-18T07:19:06.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:05.070398+0000 mgr.smithi084.sihbqg (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:06.647 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Added host 'smithi084' 2024-01-18T07:19:06.647 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.645+0000 7f4ed7fff700 1 -- 172.21.15.84:0/2761447338 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f4ef0101f70 con 0x7f4edc02fdb0 2024-01-18T07:19:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 msgr2=0x7f4edc032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 0x7f4edc032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4ee40096a0 tx=0x7f4ee4006b40).stop 2024-01-18T07:19:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 msgr2=0x7f4ef010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f4ee000e040 tx=0x7f4ee0006ee0).stop 2024-01-18T07:19:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 shutdown_connections 2024-01-18T07:19:06.650 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f4edc02fdb0 0x7f4edc032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:06.650 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 --2- 172.21.15.84:0/2761447338 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ef00703b0 0x7f4ef010ccf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:06.650 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 >> 172.21.15.84:0/2761447338 conn(0x7f4ef00f63c0 msgr2=0x7f4ef00f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:06.650 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 shutdown_connections 2024-01-18T07:19:06.650 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:06.647+0000 7f4ef5d4c700 1 -- 172.21.15.84:0/2761447338 wait complete. 2024-01-18T07:19:07.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:05.991504+0000 mgr.smithi084.sihbqg (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:07.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:06.646153+0000 mon.smithi084 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:07.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:06.646674+0000 mon.smithi084 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:08.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:06.646427+0000 mgr.smithi084.sihbqg (mgr.14114) 9 : cephadm [INF] Added host smithi084 2024-01-18T07:19:08.407 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mon service with default placement... 2024-01-18T07:19:08.407 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f7b12a700 1 Processor -- start 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f7b12a700 1 -- start start 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f7b12a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f7b12a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f740fcbe0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f78ec6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f78ec6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40694/0 (socket says 172.21.15.84:40694) 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f78ec6700 1 -- 172.21.15.84:0/3671388399 learned_addr learned my addr 172.21.15.84:0/3671388399 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f78ec6700 1 -- 172.21.15.84:0/3671388399 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f740fcd20 con 0x7f7f740fc2d0 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f78ec6700 1 --2- 172.21.15.84:0/3671388399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7f6800b5f0 tx=0x7f7f68006ee0).ready entity=mon.0 client_cookie=ff466aa2d2f0141a server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.725+0000 7f7f737fe700 1 -- 172.21.15.84:0/3671388399 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6800d600 con 0x7f7f740fc2d0 2024-01-18T07:19:09.876 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f737fe700 1 -- 172.21.15.84:0/3671388399 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7f6800dbd0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f737fe700 1 -- 172.21.15.84:0/3671388399 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f68011c00 con 0x7f7f740fc2d0 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3671388399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 msgr2=0x7f7f740fc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3671388399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f7f6800b5f0 tx=0x7f7f68006ee0).stop 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3671388399 shutdown_connections 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3671388399 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f740fc6f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3671388399 >> 172.21.15.84:0/3671388399 conn(0x7f7f740f5b40 msgr2=0x7f7f740f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3671388399 shutdown_connections 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.726+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3671388399 wait complete. 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f7b12a700 1 Processor -- start 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f7b12a700 1 -- start start 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f7b12a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:09.877 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f7b12a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7410d0a0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f78ec6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f78ec6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40700/0 (socket says 172.21.15.84:40700) 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f78ec6700 1 -- 172.21.15.84:0/3903789544 learned_addr learned my addr 172.21.15.84:0/3903789544 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.727+0000 7f7f78ec6700 1 -- 172.21.15.84:0/3903789544 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6800b200 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f78ec6700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7f6800b1d0 tx=0x7f7f68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6801b410 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7f6801b9e0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7410d2a0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f6801a840 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7410d6b0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.728+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7f6800f070 con 0x7f7f740fc2d0 2024-01-18T07:19:09.878 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.729+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f74045ad0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.729+0000 7f7f717fa700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 0x7f7f58032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:09.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.729+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7f680411d0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.729+0000 7f7f73fff700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 0x7f7f58032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:09.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.729+0000 7f7f73fff700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 0x7f7f58032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7f600096a0 tx=0x7f7f60006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:09.879 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.733+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f680228b0 con 0x7f7f740fc2d0 2024-01-18T07:19:09.912 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.910+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f7410dff0 con 0x7f7f5802fdb0 2024-01-18T07:19:09.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.915+0000 7f7f717fa700 1 -- 172.21.15.84:0/3903789544 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7f7410dff0 con 0x7f7f5802fdb0 2024-01-18T07:19:09.916 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 msgr2=0x7f7f58032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 0x7f7f58032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7f600096a0 tx=0x7f7f60006b40).stop 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 msgr2=0x7f7f7410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f7f6800b1d0 tx=0x7f7f68006ee0).stop 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 shutdown_connections 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f7f5802fdb0 0x7f7f58032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 --2- 172.21.15.84:0/3903789544 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f740fc2d0 0x7f7f7410cbb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 >> 172.21.15.84:0/3903789544 conn(0x7f7f740f5b40 msgr2=0x7f7f740f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 shutdown_connections 2024-01-18T07:19:09.919 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:09.916+0000 7f7f7b12a700 1 -- 172.21.15.84:0/3903789544 wait complete. 2024-01-18T07:19:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:09.913449+0000 mgr.smithi084.sihbqg (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:09.914154+0000 mgr.smithi084.sihbqg (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-18T07:19:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:09.916497+0000 mon.smithi084 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:12.365 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying mgr service with default placement... 2024-01-18T07:19:12.365 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-18T07:19:13.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.839+0000 7f723898f700 1 Processor -- start 2024-01-18T07:19:13.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.839+0000 7f723898f700 1 -- start start 2024-01-18T07:19:13.957 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.839+0000 7f723898f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.839+0000 7f723898f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72340fb490 con 0x7f72340fab80 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40708/0 (socket says 172.21.15.84:40708) 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723259c700 1 -- 172.21.15.84:0/3877032968 learned_addr learned my addr 172.21.15.84:0/3877032968 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723259c700 1 -- 172.21.15.84:0/3877032968 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340fb5d0 con 0x7f72340fab80 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723259c700 1 --2- 172.21.15.84:0/3877032968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f722800b5f0 tx=0x7f7228006ee0).ready entity=mon.0 client_cookie=3599043e4b67efd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723159a700 1 -- 172.21.15.84:0/3877032968 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722800d600 con 0x7f72340fab80 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723159a700 1 -- 172.21.15.84:0/3877032968 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f722800dbd0 con 0x7f72340fab80 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.840+0000 7f723159a700 1 -- 172.21.15.84:0/3877032968 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7228011c00 con 0x7f72340fab80 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- 172.21.15.84:0/3877032968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 msgr2=0x7f72340fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 --2- 172.21.15.84:0/3877032968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f722800b5f0 tx=0x7f7228006ee0).stop 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- 172.21.15.84:0/3877032968 shutdown_connections 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 --2- 172.21.15.84:0/3877032968 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f72340fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:13.958 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- 172.21.15.84:0/3877032968 >> 172.21.15.84:0/3877032968 conn(0x7f72340f63c0 msgr2=0x7f72340f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- 172.21.15.84:0/3877032968 shutdown_connections 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- 172.21.15.84:0/3877032968 wait complete. 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 Processor -- start 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- start start 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.841+0000 7f723898f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234109df0 con 0x7f72340fab80 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40720/0 (socket says 172.21.15.84:40720) 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723259c700 1 -- 172.21.15.84:0/4268966447 learned_addr learned my addr 172.21.15.84:0/4268966447 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723259c700 1 -- 172.21.15.84:0/4268966447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722800b200 con 0x7f72340fab80 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723259c700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f722800b1d0 tx=0x7f7228006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722801b410 con 0x7f72340fab80 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f722801b9e0 con 0x7f72340fab80 2024-01-18T07:19:13.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f722801a840 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7234109ff0 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.842+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723410a400 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f722800f070 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f7222ffd700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 0x7f7218032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7228041310 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7234045ad0 con 0x7f72340fab80 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f7231d9b700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 0x7f7218032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.843+0000 7f7231d9b700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 0x7f7218032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f72240096a0 tx=0x7f7224006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:13.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:13.847+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7228016080 con 0x7f72340fab80 2024-01-18T07:19:14.028 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.027+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f7234103840 con 0x7f721802fd50 2024-01-18T07:19:14.032 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.030+0000 7f7222ffd700 1 -- 172.21.15.84:0/4268966447 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7234103840 con 0x7f721802fd50 2024-01-18T07:19:14.032 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-18T07:19:14.034 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 msgr2=0x7f7218032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 0x7f7218032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f72240096a0 tx=0x7f7224006b40).stop 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 msgr2=0x7f7234109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f722800b1d0 tx=0x7f7228006ee0).stop 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 shutdown_connections 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f721802fd50 0x7f7218032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 --2- 172.21.15.84:0/4268966447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f72340fab80 0x7f7234109900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 >> 172.21.15.84:0/4268966447 conn(0x7f72340f63c0 msgr2=0x7f72340f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 shutdown_connections 2024-01-18T07:19:14.035 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:14.032+0000 7f723898f700 1 -- 172.21.15.84:0/4268966447 wait complete. 2024-01-18T07:19:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:14.029468+0000 mgr.smithi084.sihbqg (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:14.029999+0000 mgr.smithi084.sihbqg (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-18T07:19:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:14.032147+0000 mon.smithi084 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:15.406 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying crash service with default placement... 2024-01-18T07:19:15.406 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea96462700 1 Processor -- start 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea96462700 1 -- start start 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea96462700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea96462700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900fee10 con 0x7fea900703f0 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea8ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea8ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40732/0 (socket says 172.21.15.84:40732) 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.807+0000 7fea8ffff700 1 -- 172.21.15.84:0/3976641201 learned_addr learned my addr 172.21.15.84:0/3976641201 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:15.812 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.808+0000 7fea8ffff700 1 -- 172.21.15.84:0/3976641201 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900fef50 con 0x7fea900703f0 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.808+0000 7fea8ffff700 1 --2- 172.21.15.84:0/3976641201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fea8400b5f0 tx=0x7fea84006ee0).ready entity=mon.0 client_cookie=b33c80e24573ee04 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.808+0000 7fea8effd700 1 -- 172.21.15.84:0/3976641201 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea8400d600 con 0x7fea900703f0 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.808+0000 7fea8effd700 1 -- 172.21.15.84:0/3976641201 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fea8400dbd0 con 0x7fea900703f0 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.808+0000 7fea8effd700 1 -- 172.21.15.84:0/3976641201 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea84011c00 con 0x7fea900703f0 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- 172.21.15.84:0/3976641201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 msgr2=0x7fea900fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 --2- 172.21.15.84:0/3976641201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fea8400b5f0 tx=0x7fea84006ee0).stop 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- 172.21.15.84:0/3976641201 shutdown_connections 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 --2- 172.21.15.84:0/3976641201 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea900fe920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- 172.21.15.84:0/3976641201 >> 172.21.15.84:0/3976641201 conn(0x7fea900f63c0 msgr2=0x7fea900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- 172.21.15.84:0/3976641201 shutdown_connections 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- 172.21.15.84:0/3976641201 wait complete. 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 Processor -- start 2024-01-18T07:19:15.813 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.809+0000 7fea96462700 1 -- start start 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea96462700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea96462700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9010aeb0 con 0x7fea900703f0 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40748/0 (socket says 172.21.15.84:40748) 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8ffff700 1 -- 172.21.15.84:0/3964176938 learned_addr learned my addr 172.21.15.84:0/3964176938 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:15.814 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8ffff700 1 -- 172.21.15.84:0/3964176938 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8400b200 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8ffff700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fea8400b190 tx=0x7fea84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea8401b410 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9010b0b0 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fea8401b9e0 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9010b4c0 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.810+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea8401a880 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.811+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fea8400f070 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.811+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea90045ad0 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.811+0000 7fea8cff9700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 0x7fea7c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.811+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fea840420d0 con 0x7fea900703f0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.811+0000 7fea8f7fe700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 0x7fea7c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:15.815 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.812+0000 7fea8f7fe700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 0x7fea7c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fea800096a0 tx=0x7fea80006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:15.817 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:15.816+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea84025460 con 0x7fea900703f0 2024-01-18T07:19:16.009 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.007+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fea9010be00 con 0x7fea7c02fdb0 2024-01-18T07:19:16.013 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.012+0000 7fea8cff9700 1 -- 172.21.15.84:0/3964176938 <== mgr.14114 v2:172.21.15.84:6800/2693695158 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fea9010be00 con 0x7fea7c02fdb0 2024-01-18T07:19:16.014 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 msgr2=0x7fea7c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 0x7fea7c032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fea800096a0 tx=0x7fea80006b40).stop 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 msgr2=0x7fea9010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fea8400b190 tx=0x7fea84006ee0).stop 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 shutdown_connections 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7fea7c02fdb0 0x7fea7c032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:16.016 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 --2- 172.21.15.84:0/3964176938 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fea900703f0 0x7fea9010a9c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:16.017 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 >> 172.21.15.84:0/3964176938 conn(0x7fea900f63c0 msgr2=0x7fea900f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:16.017 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 shutdown_connections 2024-01-18T07:19:16.017 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.014+0000 7fea96462700 1 -- 172.21.15.84:0/3964176938 wait complete. 2024-01-18T07:19:16.548 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling mgr prometheus module... 2024-01-18T07:19:16.548 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.940+0000 7ff2215b9700 1 Processor -- start 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff2215b9700 1 -- start start 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff2215b9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff2215b9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0fcbc0 con 0x7ff21c070430 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff21affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff21affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33086/0 (socket says 172.21.15.84:33086) 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.941+0000 7ff21affd700 1 -- 172.21.15.84:0/2310413158 learned_addr learned my addr 172.21.15.84:0/2310413158 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.942+0000 7ff21affd700 1 -- 172.21.15.84:0/2310413158 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0fcd00 con 0x7ff21c070430 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.942+0000 7ff21affd700 1 --2- 172.21.15.84:0/2310413158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff20c00b5f0 tx=0x7ff20c006ee0).ready entity=mon.0 client_cookie=b4ef5ba249176b42 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.942+0000 7ff219ffb700 1 -- 172.21.15.84:0/2310413158 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff20c00d600 con 0x7ff21c070430 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.942+0000 7ff219ffb700 1 -- 172.21.15.84:0/2310413158 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff20c00dc10 con 0x7ff21c070430 2024-01-18T07:19:16.944 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.942+0000 7ff219ffb700 1 -- 172.21.15.84:0/2310413158 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff20c011cd0 con 0x7ff21c070430 2024-01-18T07:19:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 -- 172.21.15.84:0/2310413158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 msgr2=0x7ff21c0fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2310413158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7ff20c00b5f0 tx=0x7ff20c006ee0).stop 2024-01-18T07:19:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 -- 172.21.15.84:0/2310413158 shutdown_connections 2024-01-18T07:19:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2310413158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c0fc6d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 -- 172.21.15.84:0/2310413158 >> 172.21.15.84:0/2310413158 conn(0x7ff21c0f63c0 msgr2=0x7ff21c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.943+0000 7ff2215b9700 1 -- 172.21.15.84:0/2310413158 shutdown_connections 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.944+0000 7ff2215b9700 1 -- 172.21.15.84:0/2310413158 wait complete. 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.944+0000 7ff2215b9700 1 Processor -- start 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.944+0000 7ff2215b9700 1 -- start start 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.944+0000 7ff2215b9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.944+0000 7ff2215b9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c111550 con 0x7ff21c070430 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff21affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff21affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33094/0 (socket says 172.21.15.84:33094) 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff21affd700 1 -- 172.21.15.84:0/2168194654 learned_addr learned my addr 172.21.15.84:0/2168194654 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:16.947 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff21affd700 1 -- 172.21.15.84:0/2168194654 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c00b200 con 0x7ff21c070430 2024-01-18T07:19:16.948 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff21affd700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff20c00b1d0 tx=0x7ff20c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:16.948 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff20c011df0 con 0x7ff21c070430 2024-01-18T07:19:16.948 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff20c01b8c0 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff20c01a830 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c111750 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.945+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c111b60 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.946+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff20c00f070 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.946+0000 7ff2037fe700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 0x7ff208032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.946+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff20c01b410 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.946+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c059100 con 0x7ff21c070430 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.946+0000 7ff21a7fc700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 0x7ff208032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:16.949 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.947+0000 7ff21a7fc700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 0x7ff208032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff2100096a0 tx=0x7ff210006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:16.952 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:16.951+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff20c016070 con 0x7ff21c070430 2024-01-18T07:19:17.152 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:17.150+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7ff21c0fdab0 con 0x7ff21c070430 2024-01-18T07:19:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:16.010240+0000 mgr.smithi084.sihbqg (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:16.010929+0000 mgr.smithi084.sihbqg (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-18T07:19:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:16.013838+0000 mon.smithi084 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:16.554618+0000 mon.smithi084 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:16.777255+0000 mon.smithi084 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:18.020 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.016+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7ff20c024790 con 0x7ff21c070430 2024-01-18T07:19:18.020 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.017+0000 7ff2037fe700 1 -- 172.21.15.84:0/2168194654 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7ff20c005b70 con 0x7ff21c070430 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 msgr2=0x7ff208032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 0x7ff208032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff2100096a0 tx=0x7ff210006b40).stop 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 msgr2=0x7ff21c111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff20c00b1d0 tx=0x7ff20c006ee0).stop 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 shutdown_connections 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7ff20802fdb0 0x7ff208032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 --2- 172.21.15.84:0/2168194654 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff21c070430 0x7ff21c111060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.019+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 >> 172.21.15.84:0/2168194654 conn(0x7ff21c0f63c0 msgr2=0x7ff21c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.020+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 shutdown_connections 2024-01-18T07:19:18.021 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.020+0000 7ff2215b9700 1 -- 172.21.15.84:0/2168194654 wait complete. 2024-01-18T07:19:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:17.152619+0000 mon.smithi084 (mon.0) 61 : audit [INF] from='client.? 172.21.15.84:0/2168194654' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-18T07:19:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:17.415504+0000 mon.smithi084 (mon.0) 62 : audit [INF] from='mgr.14114 172.21.15.84:0/4284532851' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:18.581 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying prometheus service with default placement... 2024-01-18T07:19:18.581 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-18T07:19:18.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4f105700 1 Processor -- start 2024-01-18T07:19:18.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4f105700 1 -- start start 2024-01-18T07:19:18.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4f105700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:18.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4f105700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480fb490 con 0x7f8b480fab80 2024-01-18T07:19:18.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4cea1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4cea1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33118/0 (socket says 172.21.15.84:33118) 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.962+0000 7f8b4cea1700 1 -- 172.21.15.84:0/2742235300 learned_addr learned my addr 172.21.15.84:0/2742235300 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4cea1700 1 -- 172.21.15.84:0/2742235300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480fb5d0 con 0x7f8b480fab80 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4cea1700 1 --2- 172.21.15.84:0/2742235300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8b3c00b5f0 tx=0x7f8b3c006ee0).ready entity=mon.0 client_cookie=dccaee46aa9aa87e server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b477fe700 1 -- 172.21.15.84:0/2742235300 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c00f810 con 0x7f8b480fab80 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b477fe700 1 -- 172.21.15.84:0/2742235300 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b3c00fe20 con 0x7f8b480fab80 2024-01-18T07:19:18.965 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b477fe700 1 -- 172.21.15.84:0/2742235300 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c00e3c0 con 0x7f8b480fab80 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4f105700 1 -- 172.21.15.84:0/2742235300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 msgr2=0x7f8b480fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4f105700 1 --2- 172.21.15.84:0/2742235300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f8b3c00b5f0 tx=0x7f8b3c006ee0).stop 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4f105700 1 -- 172.21.15.84:0/2742235300 shutdown_connections 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4f105700 1 --2- 172.21.15.84:0/2742235300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b480fafa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.963+0000 7f8b4f105700 1 -- 172.21.15.84:0/2742235300 >> 172.21.15.84:0/2742235300 conn(0x7f8b480f63c0 msgr2=0x7f8b480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 -- 172.21.15.84:0/2742235300 shutdown_connections 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 -- 172.21.15.84:0/2742235300 wait complete. 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 Processor -- start 2024-01-18T07:19:18.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 -- start start 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.964+0000 7f8b4f105700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b48108ce0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4cea1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4cea1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33128/0 (socket says 172.21.15.84:33128) 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4cea1700 1 -- 172.21.15.84:0/828414214 learned_addr learned my addr 172.21.15.84:0/828414214 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4cea1700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c00b200 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4cea1700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8b3c00b1d0 tx=0x7f8b3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c00e7e0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b3c00edf0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b48108ee0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c019d50 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.965+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b481092f0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f8b3c0233f0 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b457fa700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8b3c041a10 con 0x7f8b480fab80 2024-01-18T07:19:18.969 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b47fff700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2693695158 2024-01-18T07:19:18.970 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:19:18.970 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.966+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48045ad0 con 0x7f8b480fab80 2024-01-18T07:19:18.971 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:18.970+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b3c014070 con 0x7f8b480fab80 2024-01-18T07:19:19.155 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:19.154+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b481026e0 con 0x7f8b2c034210 2024-01-18T07:19:19.167 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:19.167+0000 7f8b47fff700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2693695158 2024-01-18T07:19:19.167 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:19.167+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T07:19:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:18.018590+0000 mon.smithi084 (mon.0) 63 : audit [INF] from='client.? 172.21.15.84:0/2168194654' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-18T07:19:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:18.018635+0000 mon.smithi084 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi084.sihbqg(active, since 17s) 2024-01-18T07:19:19.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:19.568+0000 7f8b47fff700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2693695158 2024-01-18T07:19:19.569 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:19.568+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T07:19:20.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:20.369+0000 7f8b47fff700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2693695158 2024-01-18T07:19:20.370 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:20.369+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T07:19:21.971 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:21.970+0000 7f8b47fff700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2693695158 2024-01-18T07:19:21.971 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:21.970+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T07:19:22.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:22.962+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f8b3c021db0 con 0x7f8b480fab80 2024-01-18T07:19:22.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:22.962+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 msgr2=0x7f8b2c0366d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:19:22.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:22.962+0000 7f8b457fa700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:22.910630+0000 mon.smithi084 (mon.0) 65 : cluster [INF] Active manager daemon smithi084.sihbqg restarted 2024-01-18T07:19:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:22.911011+0000 mon.smithi084 (mon.0) 66 : cluster [INF] Activating manager daemon smithi084.sihbqg 2024-01-18T07:19:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:22.913213+0000 mon.smithi084 (mon.0) 67 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-18T07:19:23.967 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.966+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f8b3c023a80 con 0x7f8b480fab80 2024-01-18T07:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.966+0000 7f8b457fa700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 0x7f8b2c03a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.966+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 --> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b481026e0 con 0x7f8b2c037fe0 2024-01-18T07:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.966+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 0x7f8b2c03a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:23.968 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.967+0000 7f8b47fff700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 0x7f8b2c03a3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8b34006fd0 tx=0x7f8b34006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.973+0000 7f8b457fa700 1 -- 172.21.15.84:0/828414214 <== mgr.14142 v2:172.21.15.84:6800/2270362783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8b481026e0 con 0x7f8b2c037fe0 2024-01-18T07:19:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-18T07:19:23.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 msgr2=0x7f8b2c03a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:23.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 0x7f8b2c03a3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8b34006fd0 tx=0x7f8b34006d30).stop 2024-01-18T07:19:23.976 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 msgr2=0x7f8b481087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f8b3c00b1d0 tx=0x7f8b3c006ee0).stop 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 shutdown_connections 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8b2c037fe0 0x7f8b2c03a3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b480fab80 0x7f8b481087f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 --2- 172.21.15.84:0/828414214 >> [v2:172.21.15.84:6800/2693695158,v1:172.21.15.84:6801/2693695158] conn(0x7f8b2c034210 0x7f8b2c0366d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 >> 172.21.15.84:0/828414214 conn(0x7f8b480f63c0 msgr2=0x7f8b480ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 shutdown_connections 2024-01-18T07:19:23.977 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:23.975+0000 7f8b4f105700 1 -- 172.21.15.84:0/828414214 wait complete. 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:22.964448+0000 mon.smithi084 (mon.0) 68 : cluster [DBG] mgrmap e10: smithi084.sihbqg(active, starting, since 0.0535212s) 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:22.967044+0000 mon.smithi084 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:22.967631+0000 mon.smithi084 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi084.sihbqg", "id": "smithi084.sihbqg"}]: dispatch 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:22.968432+0000 mon.smithi084 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:22.968651+0000 mon.smithi084 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:22.968871+0000 mon.smithi084 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T07:19:24.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:22.982174+0000 mon.smithi084 (mon.0) 74 : cluster [INF] Manager daemon smithi084.sihbqg is now available 2024-01-18T07:19:24.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:23.082474+0000 mon.smithi084 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:24.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:23.084154+0000 mon.smithi084 (mon.0) 76 : audit [DBG] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:24.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:23.098798+0000 mon.smithi084 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:19:24.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:23.101311+0000 mon.smithi084 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:19:24.346 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying grafana service with default placement... 2024-01-18T07:19:24.346 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-18T07:19:24.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.957+0000 7fbe36e1a700 1 Processor -- start 2024-01-18T07:19:24.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.957+0000 7fbe36e1a700 1 -- start start 2024-01-18T07:19:24.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.957+0000 7fbe36e1a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:24.959 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.957+0000 7fbe36e1a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300fc5c0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.957+0000 7fbe34bb6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe34bb6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33186/0 (socket says 172.21.15.84:33186) 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe34bb6700 1 -- 172.21.15.84:0/3296083128 learned_addr learned my addr 172.21.15.84:0/3296083128 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe34bb6700 1 -- 172.21.15.84:0/3296083128 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300fc700 con 0x7fbe300fbcb0 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe34bb6700 1 --2- 172.21.15.84:0/3296083128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbe2400b5f0 tx=0x7fbe24006ee0).ready entity=mon.0 client_cookie=8b90768e446f49bf server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe2f7fe700 1 -- 172.21.15.84:0/3296083128 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2400d600 con 0x7fbe300fbcb0 2024-01-18T07:19:24.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe2f7fe700 1 -- 172.21.15.84:0/3296083128 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe2400dc10 con 0x7fbe300fbcb0 2024-01-18T07:19:24.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.958+0000 7fbe2f7fe700 1 -- 172.21.15.84:0/3296083128 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe24011cd0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- 172.21.15.84:0/3296083128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 msgr2=0x7fbe300fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:24.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/3296083128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbe2400b5f0 tx=0x7fbe24006ee0).stop 2024-01-18T07:19:24.962 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- 172.21.15.84:0/3296083128 shutdown_connections 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/3296083128 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe300fc0d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- 172.21.15.84:0/3296083128 >> 172.21.15.84:0/3296083128 conn(0x7fbe300f74b0 msgr2=0x7fbe300f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- 172.21.15.84:0/3296083128 shutdown_connections 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- 172.21.15.84:0/3296083128 wait complete. 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 Processor -- start 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.959+0000 7fbe36e1a700 1 -- start start 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe36e1a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe36e1a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3010af40 con 0x7fbe300fbcb0 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe34bb6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe34bb6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33196/0 (socket says 172.21.15.84:33196) 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe34bb6700 1 -- 172.21.15.84:0/1245482782 learned_addr learned my addr 172.21.15.84:0/1245482782 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe34bb6700 1 -- 172.21.15.84:0/1245482782 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2400b200 con 0x7fbe300fbcb0 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.960+0000 7fbe34bb6700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbe2400dfb0 tx=0x7fbe24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:24.963 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe24011df0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe2401b8c0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3010b140 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2401a8c0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3010b550 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.961+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fbe2400f070 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.962+0000 7fbe2d7fa700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 0x7fbe140321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.962+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbe24040f40 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.962+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30045ad0 con 0x7fbe300fbcb0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.962+0000 7fbe2ffff700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 0x7fbe140321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:24.964 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.962+0000 7fbe2ffff700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 0x7fbe140321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe1c0096a0 tx=0x7fbe1c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:24.966 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:24.966+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe24010be0 con 0x7fbe300fbcb0 2024-01-18T07:19:25.002 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:23.969071+0000 mon.smithi084 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi084.sihbqg(active, since 1.05815s) 2024-01-18T07:19:25.003 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:23.970365+0000 mgr.smithi084.sihbqg (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-18T07:19:25.003 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:23.973375+0000 mon.smithi084 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:25.003 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:24.008764+0000 mon.smithi084 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:25.009 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.008+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbe24024930 con 0x7fbe300fbcb0 2024-01-18T07:19:25.148 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.148+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 --> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe30104a00 con 0x7fbe1402fcf0 2024-01-18T07:19:25.156 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.155+0000 7fbe2d7fa700 1 -- 172.21.15.84:0/1245482782 <== mgr.14142 v2:172.21.15.84:6800/2270362783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fbe30104a00 con 0x7fbe1402fcf0 2024-01-18T07:19:25.156 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-18T07:19:25.158 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 msgr2=0x7fbe140321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:25.158 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 0x7fbe140321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe1c0096a0 tx=0x7fbe1c006b40).stop 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 msgr2=0x7fbe3010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbe2400dfb0 tx=0x7fbe24006ee0).stop 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 shutdown_connections 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fbe1402fcf0 0x7fbe140321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 --2- 172.21.15.84:0/1245482782 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300fbcb0 0x7fbe3010aa50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 >> 172.21.15.84:0/1245482782 conn(0x7fbe300f74b0 msgr2=0x7fbe300f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 shutdown_connections 2024-01-18T07:19:25.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.157+0000 7fbe36e1a700 1 -- 172.21.15.84:0/1245482782 wait complete. 2024-01-18T07:19:25.545 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying node-exporter service with default placement... 2024-01-18T07:19:25.545 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-18T07:19:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aef59e700 1 Processor -- start 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aef59e700 1 -- start start 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aef59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af00fb2e0 con 0x7f4af00fa9d0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aee59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aee59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33208/0 (socket says 172.21.15.84:33208) 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aee59c700 1 -- 172.21.15.84:0/2845738484 learned_addr learned my addr 172.21.15.84:0/2845738484 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.918+0000 7f4aee59c700 1 -- 172.21.15.84:0/2845738484 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00fb420 con 0x7f4af00fa9d0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aee59c700 1 --2- 172.21.15.84:0/2845738484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4ae000b870 tx=0x7f4ae0006f70).ready entity=mon.0 client_cookie=2fc834e7c2f2b760 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aed59a700 1 -- 172.21.15.84:0/2845738484 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae000da20 con 0x7f4af00fa9d0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aed59a700 1 -- 172.21.15.84:0/2845738484 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ae0010470 con 0x7f4af00fa9d0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aed59a700 1 -- 172.21.15.84:0/2845738484 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae000f5b0 con 0x7f4af00fa9d0 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 -- 172.21.15.84:0/2845738484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 msgr2=0x7f4af00fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 --2- 172.21.15.84:0/2845738484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4ae000b870 tx=0x7f4ae0006f70).stop 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 -- 172.21.15.84:0/2845738484 shutdown_connections 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 --2- 172.21.15.84:0/2845738484 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af00fadf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 -- 172.21.15.84:0/2845738484 >> 172.21.15.84:0/2845738484 conn(0x7f4af00f6210 msgr2=0x7f4af00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 -- 172.21.15.84:0/2845738484 shutdown_connections 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.919+0000 7f4aef59e700 1 -- 172.21.15.84:0/2845738484 wait complete. 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aef59e700 1 Processor -- start 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aef59e700 1 -- start start 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aef59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aef59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0108b50 con 0x7f4af00fa9d0 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aee59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aee59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33222/0 (socket says 172.21.15.84:33222) 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aee59c700 1 -- 172.21.15.84:0/3332880611 learned_addr learned my addr 172.21.15.84:0/3332880611 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.920+0000 7f4aee59c700 1 -- 172.21.15.84:0/3332880611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae000b480 con 0x7f4af00fa9d0 2024-01-18T07:19:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4aee59c700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4ae000b840 tx=0x7f4ae0006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae000f3d0 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0108d50 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ae000fef0 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae0019c70 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0109160 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.921+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4ae0016030 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.922+0000 7f4ae6ffd700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 0x7f4ad0032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.922+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4ae00446b0 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.922+0000 7f4aedd9b700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 0x7f4ad0032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.922+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af0059100 con 0x7f4af00fa9d0 2024-01-18T07:19:25.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.922+0000 7f4aedd9b700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 0x7f4ad0032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4ad80096a0 tx=0x7f4ad8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:25.926 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:25.926+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ae0014030 con 0x7f4af00fa9d0 2024-01-18T07:19:26.107 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.106+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 --> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af0102490 con 0x7f4ad002fe60 2024-01-18T07:19:26.113 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.112+0000 7f4ae6ffd700 1 -- 172.21.15.84:0/3332880611 <== mgr.14142 v2:172.21.15.84:6800/2270362783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4af0102490 con 0x7f4ad002fe60 2024-01-18T07:19:26.113 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-18T07:19:26.115 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 msgr2=0x7f4ad0032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:26.115 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 0x7f4ad0032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4ad80096a0 tx=0x7f4ad8006b40).stop 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 msgr2=0x7f4af0108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4ae000b840 tx=0x7f4ae0006f70).stop 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 shutdown_connections 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f4ad002fe60 0x7f4ad0032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 --2- 172.21.15.84:0/3332880611 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4af00fa9d0 0x7f4af0108660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.114+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 >> 172.21.15.84:0/3332880611 conn(0x7f4af00f6210 msgr2=0x7f4af00ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.115+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 shutdown_connections 2024-01-18T07:19:26.116 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.115+0000 7f4aef59e700 1 -- 172.21.15.84:0/3332880611 wait complete. 2024-01-18T07:19:26.285 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:25.010411+0000 mon.smithi084 (mon.0) 82 : cluster [DBG] mgrmap e12: smithi084.sihbqg(active, since 2s) 2024-01-18T07:19:26.286 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:25.156308+0000 mon.smithi084 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:26.478 INFO:teuthology.orchestra.run.smithi084.stderr:Deploying alertmanager service with default placement... 2024-01-18T07:19:26.478 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.849+0000 7f68e8084700 1 Processor -- start 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.849+0000 7f68e8084700 1 -- start start 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e8084700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e8084700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00fedd0 con 0x7f68e0070430 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e5e20700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e5e20700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37864/0 (socket says 172.21.15.84:37864) 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e5e20700 1 -- 172.21.15.84:0/442943961 learned_addr learned my addr 172.21.15.84:0/442943961 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:26.852 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e5e20700 1 -- 172.21.15.84:0/442943961 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00fef10 con 0x7f68e0070430 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e5e20700 1 --2- 172.21.15.84:0/442943961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f68d000b5f0 tx=0x7f68d0006ee0).ready entity=mon.0 client_cookie=8aa5ac945b363a30 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e4e1e700 1 -- 172.21.15.84:0/442943961 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d000d600 con 0x7f68e0070430 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e4e1e700 1 -- 172.21.15.84:0/442943961 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d000dc10 con 0x7f68e0070430 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.850+0000 7f68e4e1e700 1 -- 172.21.15.84:0/442943961 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d0011cd0 con 0x7f68e0070430 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 -- 172.21.15.84:0/442943961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 msgr2=0x7f68e00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 --2- 172.21.15.84:0/442943961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f68d000b5f0 tx=0x7f68d0006ee0).stop 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 -- 172.21.15.84:0/442943961 shutdown_connections 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 --2- 172.21.15.84:0/442943961 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fe8e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 -- 172.21.15.84:0/442943961 >> 172.21.15.84:0/442943961 conn(0x7f68e00f63c0 msgr2=0x7f68e00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 -- 172.21.15.84:0/442943961 shutdown_connections 2024-01-18T07:19:26.853 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.851+0000 7f68e8084700 1 -- 172.21.15.84:0/442943961 wait complete. 2024-01-18T07:19:26.855 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e8084700 1 Processor -- start 2024-01-18T07:19:26.855 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e8084700 1 -- start start 2024-01-18T07:19:26.855 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e8084700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e8084700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e00fa7e0 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e5e20700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e5e20700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37876/0 (socket says 172.21.15.84:37876) 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.852+0000 7f68e5e20700 1 -- 172.21.15.84:0/1835107646 learned_addr learned my addr 172.21.15.84:0/1835107646 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68e5e20700 1 -- 172.21.15.84:0/1835107646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d000b200 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68e5e20700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f68d0006800 tx=0x7f68d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d0011df0 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d001b8c0 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e00fa980 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d001a830 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.853+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e00fad90 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f68d000f070 con 0x7f68e0070430 2024-01-18T07:19:26.856 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68ce7fc700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 0x7f68c8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:26.857 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f68d0041030 con 0x7f68e0070430 2024-01-18T07:19:26.857 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e0045ad0 con 0x7f68e0070430 2024-01-18T07:19:26.857 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68e561f700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 0x7f68c8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:26.857 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.854+0000 7f68e561f700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 0x7f68c8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f68dc0096a0 tx=0x7f68dc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:26.859 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:26.858+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68d00107d0 con 0x7f68e0070430 2024-01-18T07:19:27.039 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.038+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 --> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f68e0104970 con 0x7f68c802fe60 2024-01-18T07:19:27.045 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.044+0000 7f68ce7fc700 1 -- 172.21.15.84:0/1835107646 <== mgr.14142 v2:172.21.15.84:6800/2270362783 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f68e0104970 con 0x7f68c802fe60 2024-01-18T07:19:27.045 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 msgr2=0x7f68c8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 0x7f68c8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f68dc0096a0 tx=0x7f68dc006b40).stop 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 msgr2=0x7f68e00fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f68d0006800 tx=0x7f68d0006ee0).stop 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 shutdown_connections 2024-01-18T07:19:27.047 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f68c802fe60 0x7f68c8032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:27.048 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 --2- 172.21.15.84:0/1835107646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f68e0070430 0x7f68e00fc360 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:27.048 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 >> 172.21.15.84:0/1835107646 conn(0x7f68e00f63c0 msgr2=0x7f68e00f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:27.048 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 shutdown_connections 2024-01-18T07:19:27.048 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.046+0000 7f68e8084700 1 -- 172.21.15.84:0/1835107646 wait complete. 2024-01-18T07:19:27.330 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:25.149685+0000 mgr.smithi084.sihbqg (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:27.330 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:25.150232+0000 mgr.smithi084.sihbqg (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-18T07:19:27.330 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:26.113347+0000 mon.smithi084 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:27.403 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-18T07:19:27.771 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab2de4700 1 Processor -- start 2024-01-18T07:19:27.771 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab2de4700 1 -- start start 2024-01-18T07:19:27.771 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab2de4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:27.771 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab2de4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0fb490 con 0x7ffaac0fab80 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab0b80700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab0b80700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37884/0 (socket says 172.21.15.84:37884) 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.768+0000 7ffab0b80700 1 -- 172.21.15.84:0/2102334760 learned_addr learned my addr 172.21.15.84:0/2102334760 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab0b80700 1 -- 172.21.15.84:0/2102334760 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac0fb5d0 con 0x7ffaac0fab80 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab0b80700 1 --2- 172.21.15.84:0/2102334760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ffaa000b5f0 tx=0x7ffaa0006ee0).ready entity=mon.0 client_cookie=9ed4bb4a4557b180 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffaab7fe700 1 -- 172.21.15.84:0/2102334760 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa000d600 con 0x7ffaac0fab80 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffaab7fe700 1 -- 172.21.15.84:0/2102334760 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffaa000dc10 con 0x7ffaac0fab80 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffaab7fe700 1 -- 172.21.15.84:0/2102334760 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa0011cd0 con 0x7ffaac0fab80 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab2de4700 1 -- 172.21.15.84:0/2102334760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 msgr2=0x7ffaac0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab2de4700 1 --2- 172.21.15.84:0/2102334760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ffaa000b5f0 tx=0x7ffaa0006ee0).stop 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab2de4700 1 -- 172.21.15.84:0/2102334760 shutdown_connections 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab2de4700 1 --2- 172.21.15.84:0/2102334760 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac0fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.769+0000 7ffab2de4700 1 -- 172.21.15.84:0/2102334760 >> 172.21.15.84:0/2102334760 conn(0x7ffaac0f63c0 msgr2=0x7ffaac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:27.772 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 -- 172.21.15.84:0/2102334760 shutdown_connections 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 -- 172.21.15.84:0/2102334760 wait complete. 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 Processor -- start 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 -- start start 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.770+0000 7ffab2de4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0ff320 con 0x7ffaac0fab80 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab0b80700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab0b80700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37892/0 (socket says 172.21.15.84:37892) 2024-01-18T07:19:27.773 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab0b80700 1 -- 172.21.15.84:0/1961922267 learned_addr learned my addr 172.21.15.84:0/1961922267 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:27.774 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab0b80700 1 -- 172.21.15.84:0/1961922267 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa000b200 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab0b80700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ffaa000dfb0 tx=0x7ffaa0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa0011df0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffaa001b8c0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac0ff4c0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffaa001a8c0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.771+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac0ff8d0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.772+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ffaa000f070 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.772+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaac045ad0 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.772+0000 7ffaa97fa700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 0x7ffa90032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.772+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ffaa0041070 con 0x7ffaac0fab80 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.772+0000 7ffaabfff700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 0x7ffa90032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:27.775 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.773+0000 7ffaabfff700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 0x7ffa90032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ffa980096a0 tx=0x7ffa98006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:27.777 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.777+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffaa0010be0 con 0x7ffaac0fab80 2024-01-18T07:19:27.925 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.925+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ffaac0fca10 con 0x7ffaac0fab80 2024-01-18T07:19:27.928 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.928+0000 7ffaa97fa700 1 -- 172.21.15.84:0/1961922267 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7ffaa00107d0 con 0x7ffaac0fab80 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 msgr2=0x7ffa90032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 0x7ffa90032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ffa980096a0 tx=0x7ffa98006b40).stop 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 msgr2=0x7ffaac100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ffaa000dfb0 tx=0x7ffaa0006ee0).stop 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 shutdown_connections 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7ffa9002feb0 0x7ffa90032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 --2- 172.21.15.84:0/1961922267 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffaac0fab80 0x7ffaac100ea0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 >> 172.21.15.84:0/1961922267 conn(0x7ffaac0f63c0 msgr2=0x7ffaac070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:27.931 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 shutdown_connections 2024-01-18T07:19:27.932 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:27.930+0000 7ffab2de4700 1 -- 172.21.15.84:0/1961922267 wait complete. 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:26.108036+0000 mgr.smithi084.sihbqg (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:26.108647+0000 mgr.smithi084.sihbqg (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:27.040032+0000 mgr.smithi084.sihbqg (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:27.040508+0000 mgr.smithi084.sihbqg (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:27.045085+0000 mon.smithi084 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.84:0/3973534448' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:28.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:27.928834+0000 mon.smithi084 (mon.0) 86 : audit [INF] from='client.? 172.21.15.84:0/1961922267' entity='client.admin' 2024-01-18T07:19:28.319 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc019ed700 1 Processor -- start 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc019ed700 1 -- start start 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc019ed700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc019ed700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc0fec20 con 0x7efbfc070430 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc009eb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc009eb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37906/0 (socket says 172.21.15.84:37906) 2024-01-18T07:19:28.688 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc009eb700 1 -- 172.21.15.84:0/1633771414 learned_addr learned my addr 172.21.15.84:0/1633771414 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc009eb700 1 -- 172.21.15.84:0/1633771414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc0fed60 con 0x7efbfc070430 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.684+0000 7efc009eb700 1 --2- 172.21.15.84:0/1633771414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efbf000b5f0 tx=0x7efbf0006ee0).ready entity=mon.0 client_cookie=50d34a04fbf54df3 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efbfb7fe700 1 -- 172.21.15.84:0/1633771414 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf000d600 con 0x7efbfc070430 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efbfb7fe700 1 -- 172.21.15.84:0/1633771414 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf000dc10 con 0x7efbfc070430 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efbfb7fe700 1 -- 172.21.15.84:0/1633771414 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0011cd0 con 0x7efbfc070430 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efc019ed700 1 -- 172.21.15.84:0/1633771414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 msgr2=0x7efbfc0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efc019ed700 1 --2- 172.21.15.84:0/1633771414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7efbf000b5f0 tx=0x7efbf0006ee0).stop 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efc019ed700 1 -- 172.21.15.84:0/1633771414 shutdown_connections 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efc019ed700 1 --2- 172.21.15.84:0/1633771414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc0fe730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.685+0000 7efc019ed700 1 -- 172.21.15.84:0/1633771414 >> 172.21.15.84:0/1633771414 conn(0x7efbfc0f6210 msgr2=0x7efbfc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 -- 172.21.15.84:0/1633771414 shutdown_connections 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 -- 172.21.15.84:0/1633771414 wait complete. 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 Processor -- start 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 -- start start 2024-01-18T07:19:28.689 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.686+0000 7efc019ed700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbfc10adb0 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc009eb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc009eb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37918/0 (socket says 172.21.15.84:37918) 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc009eb700 1 -- 172.21.15.84:0/3010104330 learned_addr learned my addr 172.21.15.84:0/3010104330 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc009eb700 1 -- 172.21.15.84:0/3010104330 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf000b200 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc009eb700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efbf0006a40 tx=0x7efbf0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf0011c90 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbf001b8c0 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbfc10afb0 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbf001a8c0 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.687+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbfc10b3c0 con 0x7efbfc070430 2024-01-18T07:19:28.690 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7efbf000f070 con 0x7efbfc070430 2024-01-18T07:19:28.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efbf97fa700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 0x7efbe0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:28.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7efbf0041940 con 0x7efbfc070430 2024-01-18T07:19:28.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efbfbfff700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 0x7efbe0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:28.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbfc059100 con 0x7efbfc070430 2024-01-18T07:19:28.691 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.688+0000 7efbfbfff700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 0x7efbe0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7efbe80096a0 tx=0x7efbe8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:28.693 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.692+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbf0016070 con 0x7efbfc070430 2024-01-18T07:19:28.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.842+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7efbfc045ad0 con 0x7efbfc070430 2024-01-18T07:19:28.845 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.844+0000 7efbf97fa700 1 -- 172.21.15.84:0/3010104330 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7efbf0010540 con 0x7efbfc070430 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 msgr2=0x7efbe0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 0x7efbe0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7efbe80096a0 tx=0x7efbe8006b40).stop 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 msgr2=0x7efbfc10a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efbf0006a40 tx=0x7efbf0006ee0).stop 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 shutdown_connections 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7efbe002feb0 0x7efbe0032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 --2- 172.21.15.84:0/3010104330 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbfc070430 0x7efbfc10a8c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 >> 172.21.15.84:0/3010104330 conn(0x7efbfc0f6210 msgr2=0x7efbfc0f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 shutdown_connections 2024-01-18T07:19:28.848 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:28.846+0000 7efc019ed700 1 -- 172.21.15.84:0/3010104330 wait complete. 2024-01-18T07:19:29.227 INFO:teuthology.orchestra.run.smithi084.stderr:Enabling the dashboard module... 2024-01-18T07:19:29.227 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-18T07:19:29.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.595+0000 7fd3afff2700 1 Processor -- start 2024-01-18T07:19:29.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.595+0000 7fd3afff2700 1 -- start start 2024-01-18T07:19:29.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.595+0000 7fd3afff2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:29.604 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.595+0000 7fd3afff2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80fedd0 con 0x7fd3a8070430 2024-01-18T07:19:29.604 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.595+0000 7fd3add8e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:29.604 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3add8e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37920/0 (socket says 172.21.15.84:37920) 2024-01-18T07:19:29.604 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3add8e700 1 -- 172.21.15.84:0/733693947 learned_addr learned my addr 172.21.15.84:0/733693947 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:29.604 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3add8e700 1 -- 172.21.15.84:0/733693947 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80fef10 con 0x7fd3a8070430 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3add8e700 1 --2- 172.21.15.84:0/733693947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd39800b5f0 tx=0x7fd398006ee0).ready entity=mon.0 client_cookie=21b6277574d8ecb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3acd8c700 1 -- 172.21.15.84:0/733693947 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39800d600 con 0x7fd3a8070430 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3acd8c700 1 -- 172.21.15.84:0/733693947 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd39800dc10 con 0x7fd3a8070430 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.596+0000 7fd3acd8c700 1 -- 172.21.15.84:0/733693947 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd398011cd0 con 0x7fd3a8070430 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- 172.21.15.84:0/733693947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 msgr2=0x7fd3a80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 --2- 172.21.15.84:0/733693947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fd39800b5f0 tx=0x7fd398006ee0).stop 2024-01-18T07:19:29.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- 172.21.15.84:0/733693947 shutdown_connections 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 --2- 172.21.15.84:0/733693947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a80fe8e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- 172.21.15.84:0/733693947 >> 172.21.15.84:0/733693947 conn(0x7fd3a80f63c0 msgr2=0x7fd3a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- 172.21.15.84:0/733693947 shutdown_connections 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- 172.21.15.84:0/733693947 wait complete. 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 Processor -- start 2024-01-18T07:19:29.606 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- start start 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.597+0000 7fd3afff2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a810afd0 con 0x7fd3a8070430 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3add8e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3add8e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37926/0 (socket says 172.21.15.84:37926) 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3add8e700 1 -- 172.21.15.84:0/553432178 learned_addr learned my addr 172.21.15.84:0/553432178 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3add8e700 1 -- 172.21.15.84:0/553432178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39800b200 con 0x7fd3a8070430 2024-01-18T07:19:29.607 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3add8e700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd398006800 tx=0x7fd398006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd398011df0 con 0x7fd3a8070430 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd39801b8c0 con 0x7fd3a8070430 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a810b1d0 con 0x7fd3a8070430 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39801a830 con 0x7fd3a8070430 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.598+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a810b5e0 con 0x7fd3a8070430 2024-01-18T07:19:29.608 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.599+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd39800f070 con 0x7fd3a8070430 2024-01-18T07:19:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.599+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8045ad0 con 0x7fd3a8070430 2024-01-18T07:19:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.599+0000 7fd3967fc700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 0x7fd390032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.599+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd398040ff0 con 0x7fd3a8070430 2024-01-18T07:19:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.599+0000 7fd3ad58d700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 0x7fd390032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:29.609 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.600+0000 7fd3ad58d700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 0x7fd390032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3a40096a0 tx=0x7fd3a4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:29.610 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.603+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd39806d2b0 con 0x7fd3a8070430 2024-01-18T07:19:29.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.788+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd3a80ffc00 con 0x7fd3a8070430 2024-01-18T07:19:29.847 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.846+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd39801b410 con 0x7fd3a8070430 2024-01-18T07:19:29.847 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.847+0000 7fd3967fc700 1 -- 172.21.15.84:0/553432178 <== mon.0 v2:172.21.15.84:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd398043e10 con 0x7fd3a8070430 2024-01-18T07:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 msgr2=0x7fd390032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 0x7fd390032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fd3a40096a0 tx=0x7fd3a4006b40).stop 2024-01-18T07:19:29.850 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 msgr2=0x7fd3a810aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd398006800 tx=0x7fd398006ee0).stop 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 shutdown_connections 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fd39002feb0 0x7fd390032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 --2- 172.21.15.84:0/553432178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd3a8070430 0x7fd3a810aae0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 >> 172.21.15.84:0/553432178 conn(0x7fd3a80f63c0 msgr2=0x7fd3a80f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 shutdown_connections 2024-01-18T07:19:29.851 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:29.849+0000 7fd3afff2700 1 -- 172.21.15.84:0/553432178 wait complete. 2024-01-18T07:19:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:28.845440+0000 mon.smithi084 (mon.0) 87 : audit [INF] from='client.? 172.21.15.84:0/3010104330' entity='client.admin' 2024-01-18T07:19:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:29.789351+0000 mon.smithi084 (mon.0) 88 : audit [INF] from='client.? 172.21.15.84:0/553432178' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-18T07:19:30.154 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.583+0000 7fef457c0700 1 Processor -- start 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.583+0000 7fef457c0700 1 -- start start 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.583+0000 7fef457c0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.583+0000 7fef457c0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef40070770 con 0x7fef40070350 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.583+0000 7fef3effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37960/0 (socket says 172.21.15.84:37960) 2024-01-18T07:19:30.587 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3effd700 1 -- 172.21.15.84:0/2371260820 learned_addr learned my addr 172.21.15.84:0/2371260820 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3effd700 1 -- 172.21.15.84:0/2371260820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400feeb0 con 0x7fef40070350 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3effd700 1 --2- 172.21.15.84:0/2371260820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fef3400b5f0 tx=0x7fef34006ee0).ready entity=mon.0 client_cookie=b099efc5523421ee server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3dffb700 1 -- 172.21.15.84:0/2371260820 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3400d600 con 0x7fef40070350 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3dffb700 1 -- 172.21.15.84:0/2371260820 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef3400dc10 con 0x7fef40070350 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.584+0000 7fef3dffb700 1 -- 172.21.15.84:0/2371260820 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34011cd0 con 0x7fef40070350 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 -- 172.21.15.84:0/2371260820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 msgr2=0x7fef400fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 --2- 172.21.15.84:0/2371260820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fef3400b5f0 tx=0x7fef34006ee0).stop 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 -- 172.21.15.84:0/2371260820 shutdown_connections 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 --2- 172.21.15.84:0/2371260820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fe9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 -- 172.21.15.84:0/2371260820 >> 172.21.15.84:0/2371260820 conn(0x7fef400f63c0 msgr2=0x7fef400f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 -- 172.21.15.84:0/2371260820 shutdown_connections 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.585+0000 7fef457c0700 1 -- 172.21.15.84:0/2371260820 wait complete. 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef457c0700 1 Processor -- start 2024-01-18T07:19:30.588 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef457c0700 1 -- start start 2024-01-18T07:19:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef457c0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef457c0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef400fa8c0 con 0x7fef40070350 2024-01-18T07:19:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef3effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef3effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37966/0 (socket says 172.21.15.84:37966) 2024-01-18T07:19:30.589 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef3effd700 1 -- 172.21.15.84:0/3468876369 learned_addr learned my addr 172.21.15.84:0/3468876369 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef3effd700 1 -- 172.21.15.84:0/3468876369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3400b200 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.586+0000 7fef3effd700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fef3400b1d0 tx=0x7fef34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.587+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34011df0 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.587+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef400faa60 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.587+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef3401b8c0 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.587+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3401a8a0 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.587+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef400fae70 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fef3400f070 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef40045ad0 con 0x7fef40070350 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef277fe700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 0x7fef2c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:30.590 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fef34041050 con 0x7fef40070350 2024-01-18T07:19:30.591 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef3e7fc700 1 -- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 msgr2=0x7fef2c032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:30.591 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.588+0000 7fef3e7fc700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 0x7fef2c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:19:30.593 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.592+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef34010be0 con 0x7fef40070350 2024-01-18T07:19:30.788 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.786+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fef401109c0 con 0x7fef40070350 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.787+0000 7fef277fe700 1 -- 172.21.15.84:0/3468876369 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fef34016070 con 0x7fef40070350 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "available": true, 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "active_name": "smithi084.sihbqg", 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-18T07:19:30.789 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:19:30.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.788+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 msgr2=0x7fef2c032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:19:30.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.788+0000 7fef457c0700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 0x7fef2c032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:30.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.788+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 msgr2=0x7fef400fc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:30.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.788+0000 7fef457c0700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fef3400b1d0 tx=0x7fef34006ee0).stop 2024-01-18T07:19:30.792 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 shutdown_connections 2024-01-18T07:19:30.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7fef2c02feb0 0x7fef2c032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:30.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 --2- 172.21.15.84:0/3468876369 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fef40070350 0x7fef400fc440 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:30.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 >> 172.21.15.84:0/3468876369 conn(0x7fef400f63c0 msgr2=0x7fef400f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:30.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 shutdown_connections 2024-01-18T07:19:30.793 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:30.789+0000 7fef457c0700 1 -- 172.21.15.84:0/3468876369 wait complete. 2024-01-18T07:19:31.062 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:29.848121+0000 mon.smithi084 (mon.0) 89 : audit [INF] from='client.? 172.21.15.84:0/553432178' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-18T07:19:31.063 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:29.848169+0000 mon.smithi084 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi084.sihbqg(active, since 6s) 2024-01-18T07:19:31.063 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:30.787876+0000 mon.smithi084 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.84:0/3468876369' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-18T07:19:31.145 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for the mgr to restart... 2024-01-18T07:19:31.145 INFO:teuthology.orchestra.run.smithi084.stderr:Waiting for mgr epoch 13... 2024-01-18T07:19:31.145 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.552+0000 7f8cd70d5700 1 Processor -- start 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.552+0000 7f8cd70d5700 1 -- start start 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.552+0000 7f8cd70d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.552+0000 7f8cd70d5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0072890 con 0x7f8cd0072470 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8cd4e71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8cd4e71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37974/0 (socket says 172.21.15.84:37974) 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8cd4e71700 1 -- 172.21.15.84:0/2408577821 learned_addr learned my addr 172.21.15.84:0/2408577821 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8cd4e71700 1 -- 172.21.15.84:0/2408577821 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0070f00 con 0x7f8cd0072470 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8cd4e71700 1 --2- 172.21.15.84:0/2408577821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8cc400b5f0 tx=0x7f8cc4006ee0).ready entity=mon.0 client_cookie=cd81424b0c4cedf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8ccf7fe700 1 -- 172.21.15.84:0/2408577821 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc400d600 con 0x7f8cd0072470 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8ccf7fe700 1 -- 172.21.15.84:0/2408577821 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc400dc10 con 0x7f8cd0072470 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.553+0000 7f8ccf7fe700 1 -- 172.21.15.84:0/2408577821 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4011cd0 con 0x7f8cd0072470 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 -- 172.21.15.84:0/2408577821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 msgr2=0x7f8cd0070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/2408577821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8cc400b5f0 tx=0x7f8cc4006ee0).stop 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 -- 172.21.15.84:0/2408577821 shutdown_connections 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/2408577821 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd0070a10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 -- 172.21.15.84:0/2408577821 >> 172.21.15.84:0/2408577821 conn(0x7f8cd00f74c0 msgr2=0x7f8cd00f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 -- 172.21.15.84:0/2408577821 shutdown_connections 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 -- 172.21.15.84:0/2408577821 wait complete. 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.554+0000 7f8cd70d5700 1 Processor -- start 2024-01-18T07:19:31.556 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd70d5700 1 -- start start 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd70d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd70d5700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd010d1e0 con 0x7f8cd0072470 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd4e71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd4e71700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:37984/0 (socket says 172.21.15.84:37984) 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd4e71700 1 -- 172.21.15.84:0/3132577019 learned_addr learned my addr 172.21.15.84:0/3132577019 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd4e71700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc400b200 con 0x7f8cd0072470 2024-01-18T07:19:31.558 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.555+0000 7f8cd4e71700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8cc400b1d0 tx=0x7f8cc4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.556+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4011df0 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.556+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc401b8c0 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.556+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd010d3e0 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.556+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc401a830 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.556+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd010d7f0 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8cc400f070 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccd7fa700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cb4032a80 con 0x7f8cb402ff00 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8cc4040fd0 con 0x7f8cd0072470 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccffff700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:31.559 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.557+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:19:31.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.758+0000 7f8ccffff700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:31.759 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:31.758+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-18T07:19:32.159 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:32.159+0000 7f8ccffff700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:32.160 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:32.159+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-18T07:19:32.960 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:32.960+0000 7f8ccffff700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:32.961 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:32.960+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-18T07:19:34.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:34.561+0000 7f8ccffff700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.84:6800/2270362783 2024-01-18T07:19:34.562 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:34.561+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-18T07:19:34.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:34.894+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f8cc401aa70 con 0x7f8cd0072470 2024-01-18T07:19:34.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:34.894+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 msgr2=0x7f8cb40323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:19:34.895 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:34.894+0000 7f8ccd7fa700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:35.896 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.895+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f8cc40420e0 con 0x7f8cd0072470 2024-01-18T07:19:35.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.896+0000 7f8ccd7fa700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 0x7f8cb4035f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:35.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.896+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cb4032a80 con 0x7f8cb4033b90 2024-01-18T07:19:35.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.896+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 0x7f8cb4035f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:35.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.896+0000 7f8ccffff700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 0x7f8cb4035f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8cbc006fd0 tx=0x7f8cbc006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:35.897 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.897+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f8cb4032a80 con 0x7f8cb4033b90 2024-01-18T07:19:35.904 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.903+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8cd0059100 con 0x7f8cb4033b90 2024-01-18T07:19:35.904 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.903+0000 7f8ccd7fa700 1 -- 172.21.15.84:0/3132577019 <== mgr.14162 v2:172.21.15.84:6800/2258764086 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8cd0059100 con 0x7f8cb4033b90 2024-01-18T07:19:35.904 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: { 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: "initialized": true 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: } 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.903+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 msgr2=0x7f8cb4035f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.903+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 0x7f8cb4035f80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8cbc006fd0 tx=0x7f8cbc006d70).stop 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 msgr2=0x7f8cd010ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8cc400b1d0 tx=0x7f8cc4006ee0).stop 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 shutdown_connections 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cb4033b90 0x7f8cb4035f80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cd0072470 0x7f8cd010ccf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 --2- 172.21.15.84:0/3132577019 >> [v2:172.21.15.84:6800/2270362783,v1:172.21.15.84:6801/2270362783] conn(0x7f8cb402ff00 0x7f8cb40323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:19:35.905 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 >> 172.21.15.84:0/3132577019 conn(0x7f8cd00f74c0 msgr2=0x7f8cd00f8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:35.906 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 shutdown_connections 2024-01-18T07:19:35.906 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:35.904+0000 7f8cd70d5700 1 -- 172.21.15.84:0/3132577019 wait complete. 2024-01-18T07:19:36.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:34.840614+0000 mon.smithi084 (mon.0) 92 : cluster [INF] Active manager daemon smithi084.sihbqg restarted 2024-01-18T07:19:36.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:34.841123+0000 mon.smithi084 (mon.0) 93 : cluster [INF] Activating manager daemon smithi084.sihbqg 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:34.842710+0000 mon.smithi084 (mon.0) 94 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:34.896636+0000 mon.smithi084 (mon.0) 95 : cluster [DBG] mgrmap e14: smithi084.sihbqg(active, starting, since 0.0556275s) 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:34.897661+0000 mon.smithi084 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:34.898301+0000 mon.smithi084 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi084.sihbqg", "id": "smithi084.sihbqg"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:34.899002+0000 mon.smithi084 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:34.899175+0000 mon.smithi084 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:34.899326+0000 mon.smithi084 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:34.913279+0000 mon.smithi084 (mon.0) 101 : cluster [INF] Manager daemon smithi084.sihbqg is now available 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.011397+0000 mon.smithi084 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.013172+0000 mon.smithi084 (mon.0) 103 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.029795+0000 mon.smithi084 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:19:36.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.037548+0000 mon.smithi084 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:19:36.718 INFO:teuthology.orchestra.run.smithi084.stderr:mgr epoch 13 is available 2024-01-18T07:19:36.718 INFO:teuthology.orchestra.run.smithi084.stderr:Generating a dashboard self-signed certificate... 2024-01-18T07:19:36.718 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-18T07:19:37.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.880689+0000 mon.smithi084 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:37.120 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:35.898134+0000 mon.smithi084 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi084.sihbqg(active, since 1.05713s) 2024-01-18T07:19:37.495 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde1522d700 1 Processor -- start 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde1522d700 1 -- start start 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde1522d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde1522d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100fb490 con 0x7fde100fab80 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58814/0 (socket says 172.21.15.84:58814) 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.494+0000 7fde0ed9d700 1 -- 172.21.15.84:0/3608961099 learned_addr learned my addr 172.21.15.84:0/3608961099 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde0ed9d700 1 -- 172.21.15.84:0/3608961099 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100fb5d0 con 0x7fde100fab80 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde0ed9d700 1 --2- 172.21.15.84:0/3608961099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fde0000b5f0 tx=0x7fde00006ee0).ready entity=mon.0 client_cookie=15b15f4e890b1018 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde0dd9b700 1 -- 172.21.15.84:0/3608961099 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0000d600 con 0x7fde100fab80 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde0dd9b700 1 -- 172.21.15.84:0/3608961099 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde0000dc10 con 0x7fde100fab80 2024-01-18T07:19:37.496 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde0dd9b700 1 -- 172.21.15.84:0/3608961099 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde00011cd0 con 0x7fde100fab80 2024-01-18T07:19:37.499 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde1522d700 1 -- 172.21.15.84:0/3608961099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 msgr2=0x7fde100fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:37.499 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde1522d700 1 --2- 172.21.15.84:0/3608961099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fde0000b5f0 tx=0x7fde00006ee0).stop 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde1522d700 1 -- 172.21.15.84:0/3608961099 shutdown_connections 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde1522d700 1 --2- 172.21.15.84:0/3608961099 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde100fafa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.495+0000 7fde1522d700 1 -- 172.21.15.84:0/3608961099 >> 172.21.15.84:0/3608961099 conn(0x7fde100f63c0 msgr2=0x7fde100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 -- 172.21.15.84:0/3608961099 shutdown_connections 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 -- 172.21.15.84:0/3608961099 wait complete. 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 Processor -- start 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 -- start start 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.496+0000 7fde1522d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1010d1d0 con 0x7fde100fab80 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde0ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58822/0 (socket says 172.21.15.84:58822) 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde0ed9d700 1 -- 172.21.15.84:0/4148879012 learned_addr learned my addr 172.21.15.84:0/4148879012 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde0ed9d700 1 -- 172.21.15.84:0/4148879012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0000b200 con 0x7fde100fab80 2024-01-18T07:19:37.500 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde0ed9d700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fde0000e040 tx=0x7fde00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0001b410 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde0001ba20 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0001a910 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1010d3d0 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.497+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1010d7e0 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fde0000f070 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fddf77fe700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 0x7fddfc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde00042120 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fde0e59c700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 0x7fddfc0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde100fc900 con 0x7fde100fab80 2024-01-18T07:19:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.498+0000 7fde0e59c700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 0x7fddfc0323c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fde040096a0 tx=0x7fde04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:37.503 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.502+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde00016070 con 0x7fde100fab80 2024-01-18T07:19:37.656 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.655+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fde100fcac0 con 0x7fddfc02ff00 2024-01-18T07:19:37.709 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.709+0000 7fddf77fe700 1 -- 172.21.15.84:0/4148879012 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fde100fcac0 con 0x7fddfc02ff00 2024-01-18T07:19:37.709 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-18T07:19:37.711 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 msgr2=0x7fddfc0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 0x7fddfc0323c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fde040096a0 tx=0x7fde04006b40).stop 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 msgr2=0x7fde1010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fde0000e040 tx=0x7fde00006ee0).stop 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 shutdown_connections 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fddfc02ff00 0x7fddfc0323c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 --2- 172.21.15.84:0/4148879012 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fde100fab80 0x7fde1010cce0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.710+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 >> 172.21.15.84:0/4148879012 conn(0x7fde100f63c0 msgr2=0x7fde10070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.711+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 shutdown_connections 2024-01-18T07:19:37.712 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:37.711+0000 7fde1522d700 1 -- 172.21.15.84:0/4148879012 wait complete. 2024-01-18T07:19:38.014 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.897829+0000 mgr.smithi084.sihbqg (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-18T07:19:38.014 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:35.904316+0000 mgr.smithi084.sihbqg (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-18T07:19:38.014 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:36.897710+0000 mon.smithi084 (mon.0) 108 : cluster [DBG] mgrmap e16: smithi084.sihbqg(active, since 2s) 2024-01-18T07:19:38.014 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:37.707679+0000 mon.smithi084 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:38.014 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:37.709540+0000 mon.smithi084 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:38.068 INFO:teuthology.orchestra.run.smithi084.stderr:Creating initial admin user... 2024-01-18T07:19:38.068 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmplm9gpv9n:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.446+0000 7f9b74909700 1 Processor -- start 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b74909700 1 -- start start 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b74909700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b74909700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b700fec60 con 0x7f9b700703f0 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58834/0 (socket says 172.21.15.84:58834) 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6ed9d700 1 -- 172.21.15.84:0/1486590261 learned_addr learned my addr 172.21.15.84:0/1486590261 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6ed9d700 1 -- 172.21.15.84:0/1486590261 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b700feda0 con 0x7f9b700703f0 2024-01-18T07:19:38.451 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6ed9d700 1 --2- 172.21.15.84:0/1486590261 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9b6000b5f0 tx=0x7f9b60006ee0).ready entity=mon.0 client_cookie=4185b9eddecd90f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6dd9b700 1 -- 172.21.15.84:0/1486590261 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6000d600 con 0x7f9b700703f0 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6dd9b700 1 -- 172.21.15.84:0/1486590261 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b6000dc10 con 0x7f9b700703f0 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.447+0000 7f9b6dd9b700 1 -- 172.21.15.84:0/1486590261 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b60011cd0 con 0x7f9b700703f0 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 -- 172.21.15.84:0/1486590261 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 msgr2=0x7f9b700fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 --2- 172.21.15.84:0/1486590261 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9b6000b5f0 tx=0x7f9b60006ee0).stop 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 -- 172.21.15.84:0/1486590261 shutdown_connections 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 --2- 172.21.15.84:0/1486590261 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b700fe770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 -- 172.21.15.84:0/1486590261 >> 172.21.15.84:0/1486590261 conn(0x7f9b700f6210 msgr2=0x7f9b700f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:38.452 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 -- 172.21.15.84:0/1486590261 shutdown_connections 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.448+0000 7f9b74909700 1 -- 172.21.15.84:0/1486590261 wait complete. 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b74909700 1 Processor -- start 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b74909700 1 -- start start 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b74909700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b74909700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7010ad80 con 0x7f9b700703f0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b6ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b6ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58836/0 (socket says 172.21.15.84:58836) 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b6ed9d700 1 -- 172.21.15.84:0/1339370293 learned_addr learned my addr 172.21.15.84:0/1339370293 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.449+0000 7f9b6ed9d700 1 -- 172.21.15.84:0/1339370293 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6000b200 con 0x7f9b700703f0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b6ed9d700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9b60006a40 tx=0x7f9b60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b60011c90 con 0x7f9b700703f0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b6001b8c0 con 0x7f9b700703f0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7010af80 con 0x7f9b700703f0 2024-01-18T07:19:38.453 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6001a8c0 con 0x7f9b700703f0 2024-01-18T07:19:38.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.450+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7010b390 con 0x7f9b700703f0 2024-01-18T07:19:38.454 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9b6000f070 con 0x7f9b700703f0 2024-01-18T07:19:38.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b577fe700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 0x7f9b5c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:38.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b60040a50 con 0x7f9b700703f0 2024-01-18T07:19:38.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b70045ad0 con 0x7f9b700703f0 2024-01-18T07:19:38.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b6e59c700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 0x7f9b5c0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:38.455 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.451+0000 7f9b6e59c700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 0x7f9b5c0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9b640096a0 tx=0x7f9b64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:38.456 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.455+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b60016070 con 0x7f9b700703f0 2024-01-18T07:19:38.622 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.621+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9b70058f20 con 0x7f9b5c02ff00 2024-01-18T07:19:38.839 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.838+0000 7f9b577fe700 1 -- 172.21.15.84:0/1339370293 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9b70058f20 con 0x7f9b5c02ff00 2024-01-18T07:19:38.839 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$OVHFvtI1.hz5UC/Vx1MK.OSTwJ1dZzqDheXn0WXphUMPm/Cr1mJg6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705562378, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 msgr2=0x7f9b5c0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 0x7f9b5c0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f9b640096a0 tx=0x7f9b64006b40).stop 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 msgr2=0x7f9b7010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9b60006a40 tx=0x7f9b60006ee0).stop 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 shutdown_connections 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b5c02ff00 0x7f9b5c0323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 --2- 172.21.15.84:0/1339370293 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b700703f0 0x7f9b7010a890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.840+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 >> 172.21.15.84:0/1339370293 conn(0x7f9b700f6210 msgr2=0x7f9b700f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.841+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 shutdown_connections 2024-01-18T07:19:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:38.841+0000 7f9b74909700 1 -- 172.21.15.84:0/1339370293 wait complete. 2024-01-18T07:19:39.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:37.656877+0000 mgr.smithi084.sihbqg (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:39.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:38.839527+0000 mon.smithi084 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:39.225 INFO:teuthology.orchestra.run.smithi084.stderr:Fetching dashboard port number... 2024-01-18T07:19:39.225 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpooof30fr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpuv3tqghe:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4baa936700 1 Processor -- start 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4baa936700 1 -- start start 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4baa936700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4baa936700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40fc520 con 0x7f4ba40fbc10 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4ba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4ba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58852/0 (socket says 172.21.15.84:58852) 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.596+0000 7f4ba3fff700 1 -- 172.21.15.84:0/3396652936 learned_addr learned my addr 172.21.15.84:0/3396652936 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4ba3fff700 1 -- 172.21.15.84:0/3396652936 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba40fc660 con 0x7f4ba40fbc10 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4ba3fff700 1 --2- 172.21.15.84:0/3396652936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4b9800b5f0 tx=0x7f4b98006ee0).ready entity=mon.0 client_cookie=b1bca3d4e8601e13 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4ba2ffd700 1 -- 172.21.15.84:0/3396652936 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9800d600 con 0x7f4ba40fbc10 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4ba2ffd700 1 -- 172.21.15.84:0/3396652936 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b9800dc10 con 0x7f4ba40fbc10 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4ba2ffd700 1 -- 172.21.15.84:0/3396652936 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98011cd0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 -- 172.21.15.84:0/3396652936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 msgr2=0x7f4ba40fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 --2- 172.21.15.84:0/3396652936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f4b9800b5f0 tx=0x7f4b98006ee0).stop 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 -- 172.21.15.84:0/3396652936 shutdown_connections 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 --2- 172.21.15.84:0/3396652936 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba40fc030 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 -- 172.21.15.84:0/3396652936 >> 172.21.15.84:0/3396652936 conn(0x7f4ba40f7450 msgr2=0x7f4ba40f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 -- 172.21.15.84:0/3396652936 shutdown_connections 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.597+0000 7f4baa936700 1 -- 172.21.15.84:0/3396652936 wait complete. 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4baa936700 1 Processor -- start 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4baa936700 1 -- start start 2024-01-18T07:19:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4baa936700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4baa936700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba410aed0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4ba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4ba3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58860/0 (socket says 172.21.15.84:58860) 2024-01-18T07:19:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.598+0000 7f4ba3fff700 1 -- 172.21.15.84:0/1721741409 learned_addr learned my addr 172.21.15.84:0/1721741409 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4ba3fff700 1 -- 172.21.15.84:0/1721741409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9800b200 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4ba3fff700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4b9800dfb0 tx=0x7f4b98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98011df0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b9801b8c0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba410b0d0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9801a8c0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.599+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba410b560 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4b9800f070 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4ba0ff9700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 0x7f4b900323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba4045ad0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b98041140 con 0x7f4ba40fbc10 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4ba37fe700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 0x7f4b900323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:39.603 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.600+0000 7f4ba37fe700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 0x7f4b900323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:39.605 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.604+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b98024410 con 0x7f4ba40fbc10 2024-01-18T07:19:39.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.754+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4ba40fd9f0 con 0x7f4ba40fbc10 2024-01-18T07:19:39.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.754+0000 7f4ba0ff9700 1 -- 172.21.15.84:0/1721741409 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4b9801be60 con 0x7f4ba40fbc10 2024-01-18T07:19:39.755 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 8443 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 msgr2=0x7f4b900323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 0x7f4b900323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4b940096a0 tx=0x7f4b94006b40).stop 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 msgr2=0x7f4ba410a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f4b9800dfb0 tx=0x7f4b98006ee0).stop 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 shutdown_connections 2024-01-18T07:19:39.757 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4b9002ff00 0x7f4b900323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:39.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 --2- 172.21.15.84:0/1721741409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ba40fbc10 0x7f4ba410a9e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:39.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 >> 172.21.15.84:0/1721741409 conn(0x7f4ba40f7450 msgr2=0x7f4ba40f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:39.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 shutdown_connections 2024-01-18T07:19:39.758 INFO:teuthology.orchestra.run.smithi084.stderr:/usr/bin/ceph: 2024-01-18T07:19:39.756+0000 7f4baa936700 1 -- 172.21.15.84:0/1721741409 wait complete. 2024-01-18T07:19:40.077 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:38.622961+0000 mgr.smithi084.sihbqg (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:19:40.077 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:39.755190+0000 mon.smithi084 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.84:0/1721741409' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-18T07:19:40.143 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-18T07:19:40.150 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: disabled 2024-01-18T07:19:40.150 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: systemctl is-active firewalld.service 2024-01-18T07:19:40.158 INFO:teuthology.orchestra.run.smithi084.stderr:systemctl: inactive 2024-01-18T07:19:40.158 INFO:teuthology.orchestra.run.smithi084.stderr:firewalld.service is not enabled 2024-01-18T07:19:40.158 INFO:teuthology.orchestra.run.smithi084.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-18T07:19:40.161 INFO:teuthology.orchestra.run.smithi084.stderr:Ceph Dashboard is now available at: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: User: admin 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: Password: dj81pcveq6 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr:You can access the Ceph CLI with: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: ceph telemetry on 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr:For more information see: 2024-01-18T07:19:40.162 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.163 INFO:teuthology.orchestra.run.smithi084.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-18T07:19:40.163 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:19:40.163 INFO:teuthology.orchestra.run.smithi084.stderr:Bootstrap complete. 2024-01-18T07:19:40.163 INFO:teuthology.orchestra.run.smithi084.stderr:Releasing lock 140602714339816 on /run/cephadm/bf9448d2-b5d1-11ee-95ae-87774f69a715.lock 2024-01-18T07:19:40.163 INFO:teuthology.orchestra.run.smithi084.stderr:Lock 140602714339816 released on /run/cephadm/bf9448d2-b5d1-11ee-95ae-87774f69a715.lock 2024-01-18T07:19:40.195 INFO:tasks.cephadm:Fetching config... 2024-01-18T07:19:40.195 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:19:40.196 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-18T07:19:40.254 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-18T07:19:40.254 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:19:40.254 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-18T07:19:40.313 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-18T07:19:40.313 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:19:40.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/keyring of=/dev/stdout 2024-01-18T07:19:40.381 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-18T07:19:40.382 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:19:40.382 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-18T07:19:40.440 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-18T07:19:40.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0boFXm5J0Ld8wymGwzzh3OYqzU3nfzvrXmICElPGbhsqXOVZ6sSd05Upl+viYbj5HjIQPJjwzXzLLDtn4Zw4sMGnKKl2fjrX6f5CuWLhHKTYebLF0+DkeobS9p/YcGiVcpC4Et8DH0BsBWIPdsbbgWLOvaQORvdNVmlbjC5+Jn8GoVwxMzLJ5UE5pza7eHP6HqMRkYfYCoo7QVuqEQ2MPm9SVpW3cyaKOL/+u49a0JJed/dS7qhgmSJcAbnjLne/q8V4C3CzJcXmgGYXFbjcgoP2Df66HS7SNNkCpPy2uumXyGZdAvA5HalTSQCn02hOvyTINcF+QJTiQO3SRovXHHU7nX4zgeqMh2kqTZeUFhS2wgRgo1n42w23ViDW0UlnEylw1zomtPKSYIL4jah7lmhVnuxNLRTEYZ8fo5mPVFumYfWSHR6dcNDAEvGHRN4avT9RO5Eq/xmW+rSRxAa0EJuio69R88Rwe7PrcBrXbRS+URRKwrn3u8vDJIL14Up0= ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-18T07:19:40.518 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0boFXm5J0Ld8wymGwzzh3OYqzU3nfzvrXmICElPGbhsqXOVZ6sSd05Upl+viYbj5HjIQPJjwzXzLLDtn4Zw4sMGnKKl2fjrX6f5CuWLhHKTYebLF0+DkeobS9p/YcGiVcpC4Et8DH0BsBWIPdsbbgWLOvaQORvdNVmlbjC5+Jn8GoVwxMzLJ5UE5pza7eHP6HqMRkYfYCoo7QVuqEQ2MPm9SVpW3cyaKOL/+u49a0JJed/dS7qhgmSJcAbnjLne/q8V4C3CzJcXmgGYXFbjcgoP2Df66HS7SNNkCpPy2uumXyGZdAvA5HalTSQCn02hOvyTINcF+QJTiQO3SRovXHHU7nX4zgeqMh2kqTZeUFhS2wgRgo1n42w23ViDW0UlnEylw1zomtPKSYIL4jah7lmhVnuxNLRTEYZ8fo5mPVFumYfWSHR6dcNDAEvGHRN4avT9RO5Eq/xmW+rSRxAa0EJuio69R88Rwe7PrcBrXbRS+URRKwrn3u8vDJIL14Up0= ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:19:40.529 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0boFXm5J0Ld8wymGwzzh3OYqzU3nfzvrXmICElPGbhsqXOVZ6sSd05Upl+viYbj5HjIQPJjwzXzLLDtn4Zw4sMGnKKl2fjrX6f5CuWLhHKTYebLF0+DkeobS9p/YcGiVcpC4Et8DH0BsBWIPdsbbgWLOvaQORvdNVmlbjC5+Jn8GoVwxMzLJ5UE5pza7eHP6HqMRkYfYCoo7QVuqEQ2MPm9SVpW3cyaKOL/+u49a0JJed/dS7qhgmSJcAbnjLne/q8V4C3CzJcXmgGYXFbjcgoP2Df66HS7SNNkCpPy2uumXyGZdAvA5HalTSQCn02hOvyTINcF+QJTiQO3SRovXHHU7nX4zgeqMh2kqTZeUFhS2wgRgo1n42w23ViDW0UlnEylw1zomtPKSYIL4jah7lmhVnuxNLRTEYZ8fo5mPVFumYfWSHR6dcNDAEvGHRN4avT9RO5Eq/xmW+rSRxAa0EJuio69R88Rwe7PrcBrXbRS+URRKwrn3u8vDJIL14Up0= ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-18T07:19:40.573 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0boFXm5J0Ld8wymGwzzh3OYqzU3nfzvrXmICElPGbhsqXOVZ6sSd05Upl+viYbj5HjIQPJjwzXzLLDtn4Zw4sMGnKKl2fjrX6f5CuWLhHKTYebLF0+DkeobS9p/YcGiVcpC4Et8DH0BsBWIPdsbbgWLOvaQORvdNVmlbjC5+Jn8GoVwxMzLJ5UE5pza7eHP6HqMRkYfYCoo7QVuqEQ2MPm9SVpW3cyaKOL/+u49a0JJed/dS7qhgmSJcAbnjLne/q8V4C3CzJcXmgGYXFbjcgoP2Df66HS7SNNkCpPy2uumXyGZdAvA5HalTSQCn02hOvyTINcF+QJTiQO3SRovXHHU7nX4zgeqMh2kqTZeUFhS2wgRgo1n42w23ViDW0UlnEylw1zomtPKSYIL4jah7lmhVnuxNLRTEYZ8fo5mPVFumYfWSHR6dcNDAEvGHRN4avT9RO5Eq/xmW+rSRxAa0EJuio69R88Rwe7PrcBrXbRS+URRKwrn3u8vDJIL14Up0= ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:19:40.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-18T07:19:42.278 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.274+0000 7fa5678f9700 1 -- 172.21.15.84:0/3817523289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 msgr2=0x7fa5600f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:42.278 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.274+0000 7fa5678f9700 1 --2- 172.21.15.84:0/3817523289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f9150 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa55c009fb0 tx=0x7fa55c009b30).stop 2024-01-18T07:19:42.278 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- 172.21.15.84:0/3817523289 shutdown_connections 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 --2- 172.21.15.84:0/3817523289 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f9150 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- 172.21.15.84:0/3817523289 >> 172.21.15.84:0/3817523289 conn(0x7fa5600f0b70 msgr2=0x7fa5600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- 172.21.15.84:0/3817523289 shutdown_connections 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- 172.21.15.84:0/3817523289 wait complete. 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 Processor -- start 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- start start 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.275+0000 7fa5678f9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5600f5050 con 0x7fa560060220 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa565695700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa565695700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58882/0 (socket says 172.21.15.84:58882) 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa565695700 1 -- 172.21.15.84:0/1960378335 learned_addr learned my addr 172.21.15.84:0/1960378335 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa565695700 1 -- 172.21.15.84:0/1960378335 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c005040 con 0x7fa560060220 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa565695700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa55c013510 tx=0x7fa55c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:42.279 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa55c009320 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa55c013f00 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5600f51f0 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa55c014dc0 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.276+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5600f5680 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.277+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa55c01d3d0 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.277+0000 7fa555ffb700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 0x7fa54c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.277+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa55c041f90 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.277+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa560059100 con 0x7fa560060220 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.277+0000 7fa564e94700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 0x7fa54c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:19:42.280 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.278+0000 7fa564e94700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 0x7fa54c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa5500096a0 tx=0x7fa550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:19:42.282 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.282+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa55c012920 con 0x7fa560060220 2024-01-18T07:19:42.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.440+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa5600f5e50 con 0x7fa560060220 2024-01-18T07:19:42.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.443+0000 7fa555ffb700 1 -- 172.21.15.84:0/1960378335 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa55c012330 con 0x7fa560060220 2024-01-18T07:19:42.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 msgr2=0x7fa54c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:42.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 0x7fa54c032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa5500096a0 tx=0x7fa550006b40).stop 2024-01-18T07:19:42.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 msgr2=0x7fa5600f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:19:42.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa55c013510 tx=0x7fa55c00b040).stop 2024-01-18T07:19:42.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 shutdown_connections 2024-01-18T07:19:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa54c02ff60 0x7fa54c032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 --2- 172.21.15.84:0/1960378335 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa560060220 0x7fa5600f6bd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:19:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 >> 172.21.15.84:0/1960378335 conn(0x7fa5600f0b70 msgr2=0x7fa5600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:19:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 shutdown_connections 2024-01-18T07:19:42.446 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:19:42.445+0000 7fa5678f9700 1 -- 172.21.15.84:0/1960378335 wait complete. 2024-01-18T07:19:42.829 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi106 2024-01-18T07:19:42.829 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:19:42.829 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.conf 2024-01-18T07:19:42.851 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:19:42.851 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-18T07:19:42.912 INFO:tasks.cephadm:Adding host smithi106 to orchestrator... 2024-01-18T07:19:42.912 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch host add smithi106 2024-01-18T07:19:43.072 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:41.864242+0000 mon.smithi084 (mon.0) 113 : cluster [DBG] mgrmap e17: smithi084.sihbqg(active, since 7s) 2024-01-18T07:19:43.072 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:42.443672+0000 mon.smithi084 (mon.0) 114 : audit [INF] from='client.? 172.21.15.84:0/1960378335' entity='client.admin' 2024-01-18T07:19:46.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:44.858402+0000 mon.smithi084 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:46.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:45.118976+0000 mon.smithi084 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:47.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:45.125240+0000 mgr.smithi084.sihbqg (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi084 on smithi084 2024-01-18T07:19:57.118 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:54.900079+0000 mgr.smithi084.sihbqg (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:19:57.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:56.792582+0000 mon.smithi084 (mon.0) 117 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:57.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:56.793612+0000 mon.smithi084 (mon.0) 118 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:19:57.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:56.795295+0000 mon.smithi084 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-18T07:19:57.119 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:56.795885+0000 mon.smithi084 (mon.0) 120 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:19:58.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:56.796389+0000 mgr.smithi084.sihbqg (mgr.14162) 7 : cephadm [INF] Deploying daemon crash.smithi084 on smithi084 2024-01-18T07:19:59.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:56.900315+0000 mgr.smithi084.sihbqg (mgr.14162) 8 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:19:59.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.328060+0000 mon.smithi084 (mon.0) 121 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:59.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.475743+0000 mon.smithi084 (mon.0) 122 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:59.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.478238+0000 mon.smithi084 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:19:59.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.478590+0000 mon.smithi084 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-18T07:19:59.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.482179+0000 mon.smithi084 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:20:00.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:19:58.479004+0000 mgr.smithi084.sihbqg (mgr.14162) 9 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-18T07:20:00.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:19:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:19:58.485335+0000 mgr.smithi084.sihbqg (mgr.14162) 10 : cephadm [INF] Deploying daemon grafana.smithi084 on smithi084 2024-01-18T07:20:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:19:58.900687+0000 mgr.smithi084.sihbqg (mgr.14162) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:03.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:00.901036+0000 mgr.smithi084.sihbqg (mgr.14162) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:05.283 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:02.901372+0000 mgr.smithi084.sihbqg (mgr.14162) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:05.413 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 -- 172.21.15.106:0/2997531134 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 --2- 172.21.15.106:0/2997531134 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fcb84009fb0 tx=0x7fcb84009b30).stop 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 -- 172.21.15.106:0/2997531134 shutdown_connections 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 --2- 172.21.15.106:0/2997531134 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 -- 172.21.15.106:0/2997531134 >> 172.21.15.106:0/2997531134 conn(0x7fcb900f0b70 msgr2=0x7fcb900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 -- 172.21.15.106:0/2997531134 shutdown_connections 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.410+0000 7fcb94d94700 1 -- 172.21.15.106:0/2997531134 wait complete. 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb94d94700 1 Processor -- start 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb94d94700 1 -- start start 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb94d94700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb94d94700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb90107240 con 0x7fcb900f5410 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb8e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb8e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:54790/0 (socket says 172.21.15.106:54790) 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.411+0000 7fcb8e59c700 1 -- 172.21.15.106:0/1636557315 learned_addr learned my addr 172.21.15.106:0/1636557315 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb8e59c700 1 -- 172.21.15.106:0/1636557315 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84005040 con 0x7fcb900f5410 2024-01-18T07:20:05.414 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb8e59c700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcb84013510 tx=0x7fcb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:05.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb84009320 con 0x7fcb900f5410 2024-01-18T07:20:05.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb84013f00 con 0x7fcb900f5410 2024-01-18T07:20:05.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb90060220 con 0x7fcb900f5410 2024-01-18T07:20:05.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb84014d90 con 0x7fcb900f5410 2024-01-18T07:20:05.415 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.412+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9010bd40 con 0x7fcb900f5410 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.413+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb90045ad0 con 0x7fcb900f5410 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.413+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb840095f0 con 0x7fcb900f5410 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.413+0000 7fcb7effd700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.413+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb84040fe0 con 0x7fcb900f5410 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.414+0000 7fcb8dd9b700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:05.416 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.414+0000 7fcb8dd9b700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 0x7fcb78032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcb800096a0 tx=0x7fcb80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:05.419 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.418+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb8406d2b0 con 0x7fcb900f5410 2024-01-18T07:20:05.591 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:05.590+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb900f1d70 con 0x7fcb7802ffb0 2024-01-18T07:20:06.180 INFO:teuthology.orchestra.run.smithi106.stdout:Added host 'smithi106' 2024-01-18T07:20:06.180 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.177+0000 7fcb7effd700 1 -- 172.21.15.106:0/1636557315 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fcb900f1d70 con 0x7fcb7802ffb0 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.179+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 msgr2=0x7fcb78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.179+0000 7fcb94d94700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 0x7fcb78032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcb800096a0 tx=0x7fcb80006b40).stop 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb9010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fcb84013510 tx=0x7fcb8400b040).stop 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 shutdown_connections 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 --2- 172.21.15.106:0/1636557315 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcb900f5410 0x7fcb9010b850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 >> 172.21.15.106:0/1636557315 conn(0x7fcb900f0b70 msgr2=0x7fcb900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 shutdown_connections 2024-01-18T07:20:06.181 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:06.180+0000 7fcb94d94700 1 -- 172.21.15.106:0/1636557315 wait complete. 2024-01-18T07:20:06.532 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch host ls --format=json 2024-01-18T07:20:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:04.901679+0000 mgr.smithi084.sihbqg (mgr.14162) 14 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:05.592118+0000 mgr.smithi084.sihbqg (mgr.14162) 15 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:20:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:06.178347+0000 mon.smithi084 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/53000906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 msgr2=0x7f7f580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/53000906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f580f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7f4c009fb0 tx=0x7f7f4c009b30).stop 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/53000906 shutdown_connections 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/53000906 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f580f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/53000906 >> 172.21.15.106:0/53000906 conn(0x7f7f580f1c70 msgr2=0x7f7f580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/53000906 shutdown_connections 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.643+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/53000906 wait complete. 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f5d2e3700 1 Processor -- start 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f5d2e3700 1 -- start start 2024-01-18T07:20:07.646 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f5d2e3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f5d2e3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58106c20 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f56ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f56ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:54806/0 (socket says 172.21.15.106:54806) 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f56ffd700 1 -- 172.21.15.106:0/1834618414 learned_addr learned my addr 172.21.15.106:0/1834618414 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f56ffd700 1 -- 172.21.15.106:0/1834618414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c005040 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.644+0000 7f7f56ffd700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7f4c013510 tx=0x7f7f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f4c009320 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f4c013f00 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f58106e20 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f4c014d90 con 0x7f7f580f6510 2024-01-18T07:20:07.647 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f581072b0 con 0x7f7f580f6510 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f4c0095f0 con 0x7f7f580f6510 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f58045ad0 con 0x7f7f580f6510 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.645+0000 7f7f3f7fe700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 0x7f7f40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.646+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f4c040fc0 con 0x7f7f580f6510 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.646+0000 7f7f567fc700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 0x7f7f40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:07.648 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.646+0000 7f7f567fc700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 0x7f7f40032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7f480096a0 tx=0x7f7f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:07.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.650+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f4c01d7f0 con 0x7f7f580f6510 2024-01-18T07:20:07.820 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.818+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f580f2e70 con 0x7f7f4002ff60 2024-01-18T07:20:07.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f3f7fe700 1 -- 172.21.15.106:0/1834618414 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f7f580f2e70 con 0x7f7f4002ff60 2024-01-18T07:20:07.821 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:07.821 INFO:teuthology.orchestra.run.smithi106.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "smithi106", "hostname": "smithi106", "labels": [], "status": ""}] 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 msgr2=0x7f7f40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 0x7f7f40032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7f480096a0 tx=0x7f7f48006b40).stop 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 msgr2=0x7f7f58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7f4c013510 tx=0x7f7f4c00b040).stop 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 shutdown_connections 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7f4002ff60 0x7f7f40032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 --2- 172.21.15.106:0/1834618414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7f580f6510 0x7f7f58106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.821+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 >> 172.21.15.106:0/1834618414 conn(0x7f7f580f1c70 msgr2=0x7f7f580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.822+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 shutdown_connections 2024-01-18T07:20:07.823 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:07.822+0000 7f7f5d2e3700 1 -- 172.21.15.106:0/1834618414 wait complete. 2024-01-18T07:20:08.150 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-18T07:20:08.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd crush tunables default 2024-01-18T07:20:08.177 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:20:06.178690+0000 mgr.smithi084.sihbqg (mgr.14162) 16 : cephadm [INF] Added host smithi106 2024-01-18T07:20:09.272 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:06.901924+0000 mgr.smithi084.sihbqg (mgr.14162) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:09.272 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:07.821018+0000 mgr.smithi084.sihbqg (mgr.14162) 18 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T07:20:09.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/474123856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 msgr2=0x7fb4b00f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:09.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/474123856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b00f66e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fb4a0009fb0 tx=0x7fb4a0009b30).stop 2024-01-18T07:20:09.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/474123856 shutdown_connections 2024-01-18T07:20:09.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/474123856 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b00f66e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:09.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/474123856 >> 172.21.15.84:0/474123856 conn(0x7fb4b00f1ac0 msgr2=0x7fb4b00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:09.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/474123856 shutdown_connections 2024-01-18T07:20:09.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.457+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/474123856 wait complete. 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b6bfd700 1 Processor -- start 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b6bfd700 1 -- start start 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b6bfd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b6bfd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0106a10 con 0x7fb4b00f62c0 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b5bfb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b5bfb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42342/0 (socket says 172.21.15.84:42342) 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.458+0000 7fb4b5bfb700 1 -- 172.21.15.84:0/2566683051 learned_addr learned my addr 172.21.15.84:0/2566683051 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4b5bfb700 1 -- 172.21.15.84:0/2566683051 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0005040 con 0x7fb4b00f62c0 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4b5bfb700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb4a0013510 tx=0x7fb4a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a0009320 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b0106c10 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4a0013f00 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4a0014dc0 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.459+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b01070a0 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b0045ad0 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4a001d3d0 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4a67fc700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 0x7fb49c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4a0041300 con 0x7fb4b00f62c0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4b53fa700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 0x7fb49c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:09.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.460+0000 7fb4b53fa700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 0x7fb49c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb4ac0096a0 tx=0x7fb4ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:09.469 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.468+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4a001ccf0 con 0x7fb4b00f62c0 2024-01-18T07:20:09.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.658+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb4b0107f50 con 0x7fb4b00f62c0 2024-01-18T07:20:09.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.895+0000 7fb4a67fc700 1 -- 172.21.15.84:0/2566683051 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb4a001cb10 con 0x7fb4b00f62c0 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 msgr2=0x7fb49c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 0x7fb49c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fb4ac0096a0 tx=0x7fb4ac006b40).stop 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 msgr2=0x7fb4b0106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb4a0013510 tx=0x7fb4a000b040).stop 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 shutdown_connections 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb49c02ff60 0x7fb49c032420 secure :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0x7fb4ac0096a0 tx=0x7fb4ac006b40).stop 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 --2- 172.21.15.84:0/2566683051 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb4b00f62c0 0x7fb4b0106520 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0x7fb4a0013510 tx=0x7fb4a000b040).stop 2024-01-18T07:20:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 >> 172.21.15.84:0/2566683051 conn(0x7fb4b00f1ac0 msgr2=0x7fb4b00f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:09.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 shutdown_connections 2024-01-18T07:20:09.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:20:09.898+0000 7fb4b6bfd700 1 -- 172.21.15.84:0/2566683051 wait complete. 2024-01-18T07:20:09.900 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-01-18T07:20:10.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:08.902154+0000 mgr.smithi084.sihbqg (mgr.14162) 19 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:10.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:09.660738+0000 mon.smithi084 (mon.0) 127 : audit [INF] from='client.? 172.21.15.84:0/2566683051' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-18T07:20:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:09.897530+0000 mon.smithi084 (mon.0) 128 : audit [INF] from='client.? 172.21.15.84:0/2566683051' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-18T07:20:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:09.897582+0000 mon.smithi084 (mon.0) 129 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-18T07:20:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:10.902373+0000 mgr.smithi084.sihbqg (mgr.14162) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:12.656 INFO:tasks.cephadm:Adding mon.smithi084 on smithi084 2024-01-18T07:20:12.656 INFO:tasks.cephadm:Adding mon.smithi106 on smithi106 2024-01-18T07:20:12.656 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch apply mon '2;smithi084:172.21.15.84=smithi084;smithi106:172.21.15.106=smithi106' 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 -- 172.21.15.106:0/322762573 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 msgr2=0x7f7cc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 --2- 172.21.15.106:0/322762573 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc00f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7cbc009fb0 tx=0x7f7cbc009b30).stop 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 -- 172.21.15.106:0/322762573 shutdown_connections 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 --2- 172.21.15.106:0/322762573 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc00f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 -- 172.21.15.106:0/322762573 >> 172.21.15.106:0/322762573 conn(0x7f7cc00f1c70 msgr2=0x7f7cc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:13.837 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 -- 172.21.15.106:0/322762573 shutdown_connections 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.835+0000 7f7cc8042700 1 -- 172.21.15.106:0/322762573 wait complete. 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc8042700 1 Processor -- start 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc8042700 1 -- start start 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc8042700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc8042700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc0106c20 con 0x7f7cc00f6510 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc5dde700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc5dde700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:54826/0 (socket says 172.21.15.106:54826) 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc5dde700 1 -- 172.21.15.106:0/447874068 learned_addr learned my addr 172.21.15.106:0/447874068 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc5dde700 1 -- 172.21.15.106:0/447874068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc005040 con 0x7f7cc00f6510 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.836+0000 7f7cc5dde700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7cbc013510 tx=0x7f7cbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cbc009320 con 0x7f7cc00f6510 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc0106e20 con 0x7f7cc00f6510 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cbc013f00 con 0x7f7cc00f6510 2024-01-18T07:20:13.838 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cbc014d90 con 0x7f7cc00f6510 2024-01-18T07:20:13.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc01072b0 con 0x7f7cc00f6510 2024-01-18T07:20:13.839 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cbc014420 con 0x7f7cc00f6510 2024-01-18T07:20:13.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.837+0000 7f7cb67fc700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 0x7f7cac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:13.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.838+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cbc041110 con 0x7f7cc00f6510 2024-01-18T07:20:13.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.838+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc0045ad0 con 0x7f7cc00f6510 2024-01-18T07:20:13.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.838+0000 7f7cc55dd700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 0x7f7cac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:13.840 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.838+0000 7f7cc55dd700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 0x7f7cac032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7cb00096a0 tx=0x7f7cb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:13.842 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:13.842+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cbc0095f0 con 0x7f7cc00f6510 2024-01-18T07:20:14.016 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:12.902599+0000 mgr.smithi084.sihbqg (mgr.14162) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:14.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.018+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi084:172.21.15.84=smithi084;smithi106:172.21.15.106=smithi106", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cc00f2f20 con 0x7f7cac02ff60 2024-01-18T07:20:14.023 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.022+0000 7f7cb67fc700 1 -- 172.21.15.106:0/447874068 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7cc00f2f20 con 0x7f7cac02ff60 2024-01-18T07:20:14.023 INFO:teuthology.orchestra.run.smithi106.stdout:Scheduled mon update... 2024-01-18T07:20:14.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 msgr2=0x7f7cac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:14.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 0x7f7cac032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f7cb00096a0 tx=0x7f7cb0006b40).stop 2024-01-18T07:20:14.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 msgr2=0x7f7cc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:14.024 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f7cbc013510 tx=0x7f7cbc00b040).stop 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 shutdown_connections 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7cac02ff60 0x7f7cac032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 --2- 172.21.15.106:0/447874068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7cc00f6510 0x7f7cc0106730 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 >> 172.21.15.106:0/447874068 conn(0x7f7cc00f1c70 msgr2=0x7f7cc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 shutdown_connections 2024-01-18T07:20:14.025 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:14.024+0000 7f7cc8042700 1 -- 172.21.15.106:0/447874068 wait complete. 2024-01-18T07:20:14.402 DEBUG:teuthology.orchestra.run.smithi106:mon.smithi106> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi106.service 2024-01-18T07:20:14.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:14.404 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:14.433 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:-- Logs begin at Thu 2024-01-18 07:09:06 UTC. -- 2024-01-18T07:20:15.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:14.019329+0000 mgr.smithi084.sihbqg (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi084:172.21.15.84=smithi084;smithi106:172.21.15.106=smithi106", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:20:15.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:20:14.020548+0000 mgr.smithi084.sihbqg (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi084:172.21.15.84=smithi084;smithi106:172.21.15.106=smithi106;count:2 2024-01-18T07:20:15.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:14.022799+0000 mon.smithi084 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:20:15.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:14.861638+0000 mon.smithi084 (mon.0) 131 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:20:15.032 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:20:14.862857+0000 mgr.smithi084.sihbqg (mgr.14162) 24 : cephadm [INF] Deploying daemon node-exporter.smithi084 on smithi084 2024-01-18T07:20:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 -- 172.21.15.106:0/901845899 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 msgr2=0x7f0e480f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 --2- 172.21.15.106:0/901845899 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e480f4b60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f0e3c009fb0 tx=0x7f0e3c009b30).stop 2024-01-18T07:20:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 -- 172.21.15.106:0/901845899 shutdown_connections 2024-01-18T07:20:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 --2- 172.21.15.106:0/901845899 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e480f4b60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:15.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 -- 172.21.15.106:0/901845899 >> 172.21.15.106:0/901845899 conn(0x7f0e480f0260 msgr2=0x7f0e480f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 -- 172.21.15.106:0/901845899 shutdown_connections 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.558+0000 7f0e4f231700 1 -- 172.21.15.106:0/901845899 wait complete. 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4f231700 1 Processor -- start 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4f231700 1 -- start start 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4f231700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4f231700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e48106bc0 con 0x7f0e480f4740 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4cfcd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4cfcd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50214/0 (socket says 172.21.15.106:50214) 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.559+0000 7f0e4cfcd700 1 -- 172.21.15.106:0/3722469459 learned_addr learned my addr 172.21.15.106:0/3722469459 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e4cfcd700 1 -- 172.21.15.106:0/3722469459 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c005040 con 0x7f0e480f4740 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e4cfcd700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0e3c013510 tx=0x7f0e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e3c009320 con 0x7f0e480f4740 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e3c013f00 con 0x7f0e480f4740 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e48106dc0 con 0x7f0e480f4740 2024-01-18T07:20:15.563 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e3c014d90 con 0x7f0e480f4740 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.560+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e48107250 con 0x7f0e480f4740 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.561+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e48045ad0 con 0x7f0e480f4740 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.561+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e3c0095f0 con 0x7f0e480f4740 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.561+0000 7f0e457fa700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 0x7f0e38032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.561+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e3c040fe0 con 0x7f0e480f4740 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.561+0000 7f0e47fff700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 0x7f0e38032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:15.564 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.562+0000 7f0e47fff700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 0x7f0e38032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0e340096a0 tx=0x7f0e34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:15.565 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.565+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e3c06d2b0 con 0x7f0e480f4740 2024-01-18T07:20:15.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.770+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e48108100 con 0x7f0e480f4740 2024-01-18T07:20:15.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.770+0000 7f0e457fa700 1 -- 172.21.15.106:0/3722469459 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e3c014420 con 0x7f0e480f4740 2024-01-18T07:20:15.771 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:15.771 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:15.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 msgr2=0x7f0e38032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:15.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 0x7f0e38032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0e340096a0 tx=0x7f0e34006b40).stop 2024-01-18T07:20:15.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 msgr2=0x7f0e481066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:15.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0e3c013510 tx=0x7f0e3c00b040).stop 2024-01-18T07:20:15.772 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 shutdown_connections 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0e3802ffb0 0x7f0e38032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 --2- 172.21.15.106:0/3722469459 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0e480f4740 0x7f0e481066d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 >> 172.21.15.106:0/3722469459 conn(0x7f0e480f0260 msgr2=0x7f0e480f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 shutdown_connections 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:15.772+0000 7f0e4f231700 1 -- 172.21.15.106:0/3722469459 wait complete. 2024-01-18T07:20:15.773 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:14.902821+0000 mgr.smithi084.sihbqg (mgr.14162) 25 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:15.771090+0000 mon.smithi084 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.106:0/3722469459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:17.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:17.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:18.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 -- 172.21.15.106:0/1770083552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 msgr2=0x7f90b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:18.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 --2- 172.21.15.106:0/1770083552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b00f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f90a4009fb0 tx=0x7f90a4009b30).stop 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 -- 172.21.15.106:0/1770083552 shutdown_connections 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 --2- 172.21.15.106:0/1770083552 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b00f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 -- 172.21.15.106:0/1770083552 >> 172.21.15.106:0/1770083552 conn(0x7f90b00f0b70 msgr2=0x7f90b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 -- 172.21.15.106:0/1770083552 shutdown_connections 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.282+0000 7f90b57de700 1 -- 172.21.15.106:0/1770083552 wait complete. 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.283+0000 7f90b57de700 1 Processor -- start 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.283+0000 7f90b57de700 1 -- start start 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.283+0000 7f90b57de700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:18.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.283+0000 7f90b57de700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b0060220 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90aeffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90aeffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50236/0 (socket says 172.21.15.106:50236) 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90aeffd700 1 -- 172.21.15.106:0/760795237 learned_addr learned my addr 172.21.15.106:0/760795237 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90aeffd700 1 -- 172.21.15.106:0/760795237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4005040 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90aeffd700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f90a4013510 tx=0x7f90a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a4009320 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90a4013f00 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b0060420 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a4014d90 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.284+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b0109b00 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.285+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90a40263a0 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.285+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0059100 con 0x7f90b00f5410 2024-01-18T07:20:18.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.285+0000 7f90977fe700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:18.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.285+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90a4041020 con 0x7f90b00f5410 2024-01-18T07:20:18.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.285+0000 7f90ae7fc700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:18.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.286+0000 7f90ae7fc700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 0x7f9098032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f90a00096a0 tx=0x7f90a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:18.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.290+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90a4012510 con 0x7f90b00f5410 2024-01-18T07:20:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:16.903118+0000 mgr.smithi084.sihbqg (mgr.14162) 26 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:18.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.496+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90b0045ad0 con 0x7f90b00f5410 2024-01-18T07:20:18.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.496+0000 7f90977fe700 1 -- 172.21.15.106:0/760795237 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f90a4012330 con 0x7f90b00f5410 2024-01-18T07:20:18.497 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:18.497 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 msgr2=0x7f9098032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 0x7f9098032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f90a00096a0 tx=0x7f90a0006b40).stop 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 msgr2=0x7f90b0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f90a4013510 tx=0x7f90a400b040).stop 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 shutdown_connections 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f909802ff60 0x7f9098032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 --2- 172.21.15.106:0/760795237 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f90b00f5410 0x7f90b0109450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 >> 172.21.15.106:0/760795237 conn(0x7f90b00f0b70 msgr2=0x7f90b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 shutdown_connections 2024-01-18T07:20:18.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:18.498+0000 7f90b57de700 1 -- 172.21.15.106:0/760795237 wait complete. 2024-01-18T07:20:18.500 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:18.497071+0000 mon.smithi084 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.106:0/760795237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:19.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:19.891 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:18.903438+0000 mgr.smithi084.sihbqg (mgr.14162) 27 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:21.067 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- 172.21.15.106:0/2296095479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 msgr2=0x7f972c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 --2- 172.21.15.106:0/2296095479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c0f66c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9728009fb0 tx=0x7f9728009b30).stop 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- 172.21.15.106:0/2296095479 shutdown_connections 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 --2- 172.21.15.106:0/2296095479 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c0f66c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- 172.21.15.106:0/2296095479 >> 172.21.15.106:0/2296095479 conn(0x7f972c0f1aa0 msgr2=0x7f972c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- 172.21.15.106:0/2296095479 shutdown_connections 2024-01-18T07:20:21.068 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- 172.21.15.106:0/2296095479 wait complete. 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 Processor -- start 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.066+0000 7f9732c9c700 1 -- start start 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9732c9c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9732c9c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c1069a0 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9731c9a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9731c9a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50254/0 (socket says 172.21.15.106:50254) 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9731c9a700 1 -- 172.21.15.106:0/248822349 learned_addr learned my addr 172.21.15.106:0/248822349 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9731c9a700 1 -- 172.21.15.106:0/248822349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9728005040 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9731c9a700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9728013510 tx=0x7f972800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9728009320 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9728013f00 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c106ba0 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9728014dc0 con 0x7f972c0f62a0 2024-01-18T07:20:21.069 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.067+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c107030 con 0x7f972c0f62a0 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.068+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97280095f0 con 0x7f972c0f62a0 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.068+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c045ad0 con 0x7f972c0f62a0 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.068+0000 7f97227fc700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 0x7f9718032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.068+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9728041100 con 0x7f972c0f62a0 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.069+0000 7f9731499700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 0x7f9718032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:21.071 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.069+0000 7f9731499700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 0x7f9718032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f971c0096a0 tx=0x7f971c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:21.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.073+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9728014420 con 0x7f972c0f62a0 2024-01-18T07:20:21.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.283+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f972c107ee0 con 0x7f972c0f62a0 2024-01-18T07:20:21.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.283+0000 7f97227fc700 1 -- 172.21.15.106:0/248822349 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9728010230 con 0x7f972c0f62a0 2024-01-18T07:20:21.284 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:21.284 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 msgr2=0x7f9718032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 0x7f9718032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f971c0096a0 tx=0x7f971c006b40).stop 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 msgr2=0x7f972c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9728013510 tx=0x7f972800b040).stop 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 shutdown_connections 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f971802ff60 0x7f9718032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 --2- 172.21.15.106:0/248822349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f972c0f62a0 0x7f972c1064b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 >> 172.21.15.106:0/248822349 conn(0x7f972c0f1aa0 msgr2=0x7f972c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 shutdown_connections 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:21.285+0000 7f9732c9c700 1 -- 172.21.15.106:0/248822349 wait complete. 2024-01-18T07:20:21.286 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:22.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:21.284182+0000 mon.smithi084 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.106:0/248822349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:22.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:22.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:23.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:20.903760+0000 mgr.smithi084.sihbqg (mgr.14162) 28 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:23.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- 172.21.15.106:0/3433424120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 msgr2=0x7fdc6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:23.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/3433424120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c0f55c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdc5c009fb0 tx=0x7fdc5c009b30).stop 2024-01-18T07:20:23.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- 172.21.15.106:0/3433424120 shutdown_connections 2024-01-18T07:20:23.831 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/3433424120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c0f55c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- 172.21.15.106:0/3433424120 >> 172.21.15.106:0/3433424120 conn(0x7fdc6c0f09a0 msgr2=0x7fdc6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- 172.21.15.106:0/3433424120 shutdown_connections 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- 172.21.15.106:0/3433424120 wait complete. 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 Processor -- start 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.829+0000 7fdc6bfff700 1 -- start start 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6bfff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c060220 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50264/0 (socket says 172.21.15.106:50264) 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6affd700 1 -- 172.21.15.106:0/719650279 learned_addr learned my addr 172.21.15.106:0/719650279 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6affd700 1 -- 172.21.15.106:0/719650279 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c005040 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6affd700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdc5c013510 tx=0x7fdc5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5c009320 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.832 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc5c013f00 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5c014dc0 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c060420 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.830+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c1098a0 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.831+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc5c0263a0 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.831+0000 7fdc537fe700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 0x7fdc54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.831+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc5c0095f0 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.831+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c059100 con 0x7fdc6c0f51a0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.831+0000 7fdc6a7fc700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 0x7fdc54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:23.833 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.832+0000 7fdc6a7fc700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 0x7fdc54032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdc600096a0 tx=0x7fdc60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:23.836 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:23.836+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc5c012920 con 0x7fdc6c0f51a0 2024-01-18T07:20:24.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.040+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc6c109f50 con 0x7fdc6c0f51a0 2024-01-18T07:20:24.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.041+0000 7fdc537fe700 1 -- 172.21.15.106:0/719650279 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc5c02b020 con 0x7fdc6c0f51a0 2024-01-18T07:20:24.041 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:24.041 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 msgr2=0x7fdc54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 0x7fdc54032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fdc600096a0 tx=0x7fdc60006b40).stop 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 msgr2=0x7fdc6c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fdc5c013510 tx=0x7fdc5c00b040).stop 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 shutdown_connections 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc5402ffb0 0x7fdc54032470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 --2- 172.21.15.106:0/719650279 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc6c0f51a0 0x7fdc6c1091f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 >> 172.21.15.106:0/719650279 conn(0x7fdc6c0f09a0 msgr2=0x7fdc6c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 shutdown_connections 2024-01-18T07:20:24.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:24.043+0000 7fdc6bfff700 1 -- 172.21.15.106:0/719650279 wait complete. 2024-01-18T07:20:24.044 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:25.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:22.904058+0000 mgr.smithi084.sihbqg (mgr.14162) 29 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:25.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:23.990079+0000 mon.smithi084 (mon.0) 135 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:20:25.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:24.041436+0000 mon.smithi084 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.106:0/719650279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:25.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:25.430 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:26.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:20:23.995409+0000 mgr.smithi084.sihbqg (mgr.14162) 30 : cephadm [INF] Deploying daemon prometheus.smithi084 on smithi084 2024-01-18T07:20:26.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 -- 172.21.15.106:0/2432042385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 msgr2=0x7fbe300f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:26.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 --2- 172.21.15.106:0/2432042385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe300f7d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fbe20009fb0 tx=0x7fbe20009b30).stop 2024-01-18T07:20:26.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 -- 172.21.15.106:0/2432042385 shutdown_connections 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 --2- 172.21.15.106:0/2432042385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe300f7d00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 -- 172.21.15.106:0/2432042385 >> 172.21.15.106:0/2432042385 conn(0x7fbe300f0120 msgr2=0x7fbe300f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 -- 172.21.15.106:0/2432042385 shutdown_connections 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.571+0000 7fbe367ac700 1 -- 172.21.15.106:0/2432042385 wait complete. 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe367ac700 1 Processor -- start 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe367ac700 1 -- start start 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe367ac700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe367ac700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe301069d0 con 0x7fbe300f78e0 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe357aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe357aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:34398/0 (socket says 172.21.15.106:34398) 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.572+0000 7fbe357aa700 1 -- 172.21.15.106:0/808033914 learned_addr learned my addr 172.21.15.106:0/808033914 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe357aa700 1 -- 172.21.15.106:0/808033914 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20005040 con 0x7fbe300f78e0 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe357aa700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbe20013510 tx=0x7fbe2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe20009320 con 0x7fbe300f78e0 2024-01-18T07:20:26.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe30106bd0 con 0x7fbe300f78e0 2024-01-18T07:20:26.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe20013f00 con 0x7fbe300f78e0 2024-01-18T07:20:26.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe20014dc0 con 0x7fbe300f78e0 2024-01-18T07:20:26.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.573+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe30107060 con 0x7fbe300f78e0 2024-01-18T07:20:26.575 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.574+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe200095f0 con 0x7fbe300f78e0 2024-01-18T07:20:26.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.574+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30045ad0 con 0x7fbe300f78e0 2024-01-18T07:20:26.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.574+0000 7fbe25ffb700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:26.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.574+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe20041040 con 0x7fbe300f78e0 2024-01-18T07:20:26.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.574+0000 7fbe34fa9700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:26.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.575+0000 7fbe34fa9700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 0x7fbe1c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbe2c0096a0 tx=0x7fbe2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:26.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.579+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe20014420 con 0x7fbe300f78e0 2024-01-18T07:20:26.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.784+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe30107f10 con 0x7fbe300f78e0 2024-01-18T07:20:26.785 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.785+0000 7fbe25ffb700 1 -- 172.21.15.106:0/808033914 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe20010230 con 0x7fbe300f78e0 2024-01-18T07:20:26.785 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:26.785 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 msgr2=0x7fbe1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 0x7fbe1c032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fbe2c0096a0 tx=0x7fbe2c006b40).stop 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 msgr2=0x7fbe301064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbe20013510 tx=0x7fbe2000b040).stop 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 shutdown_connections 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbe1c02ff60 0x7fbe1c032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 --2- 172.21.15.106:0/808033914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbe300f78e0 0x7fbe301064e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:26.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 >> 172.21.15.106:0/808033914 conn(0x7fbe300f0120 msgr2=0x7fbe300f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:26.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 shutdown_connections 2024-01-18T07:20:26.788 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:26.787+0000 7fbe367ac700 1 -- 172.21.15.106:0/808033914 wait complete. 2024-01-18T07:20:26.788 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:27.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:24.904298+0000 mgr.smithi084.sihbqg (mgr.14162) 31 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:27.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:26.785580+0000 mon.smithi084 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.106:0/808033914' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:28.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:28.177 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:26.904532+0000 mgr.smithi084.sihbqg (mgr.14162) 32 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:28.904930+0000 mgr.smithi084.sihbqg (mgr.14162) 33 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:32.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 -- 172.21.15.106:0/1381835500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 msgr2=0x7f74c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:32.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1381835500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c00f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f74b4009fb0 tx=0x7f74b4009b30).stop 2024-01-18T07:20:32.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 -- 172.21.15.106:0/1381835500 shutdown_connections 2024-01-18T07:20:32.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1381835500 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c00f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:32.690 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 -- 172.21.15.106:0/1381835500 >> 172.21.15.106:0/1381835500 conn(0x7f74c00f1c70 msgr2=0x7f74c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 -- 172.21.15.106:0/1381835500 shutdown_connections 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 -- 172.21.15.106:0/1381835500 wait complete. 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.663+0000 7f74c6a68700 1 Processor -- start 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c6a68700 1 -- start start 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c6a68700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c6a68700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0108e00 con 0x7f74c00f6510 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c4804700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c4804700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:34424/0 (socket says 172.21.15.106:34424) 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.664+0000 7f74c4804700 1 -- 172.21.15.106:0/1426794386 learned_addr learned my addr 172.21.15.106:0/1426794386 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74c4804700 1 -- 172.21.15.106:0/1426794386 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4005040 con 0x7f74c00f6510 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74c4804700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f74b4014000 tx=0x7f74b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b40095f0 con 0x7f74c00f6510 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74b4007e70 con 0x7f74c00f6510 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c0109000 con 0x7f74c00f6510 2024-01-18T07:20:32.691 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b4014ea0 con 0x7f74c00f6510 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.665+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c0109490 con 0x7f74c00f6510 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74b4025070 con 0x7f74c00f6510 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74bd7fa700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 0x7f74b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0045ad0 con 0x7f74c00f6510 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74b4040970 con 0x7f74c00f6510 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74bffff700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 0x7f74b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.666+0000 7f74bffff700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 0x7f74b0032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f74ac0096a0 tx=0x7f74ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:32.692 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.670+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74b40079c0 con 0x7f74c00f6510 2024-01-18T07:20:32.879 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.878+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74c010a2f0 con 0x7f74c00f6510 2024-01-18T07:20:32.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.879+0000 7f74bd7fa700 1 -- 172.21.15.106:0/1426794386 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74b400f300 con 0x7f74c00f6510 2024-01-18T07:20:32.880 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:32.880 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:32.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 msgr2=0x7f74b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:32.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 0x7f74b0032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f74ac0096a0 tx=0x7f74ac006b40).stop 2024-01-18T07:20:32.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 msgr2=0x7f74c0108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:32.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f74b4014000 tx=0x7f74b400b040).stop 2024-01-18T07:20:32.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 shutdown_connections 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f74b002ff60 0x7f74b0032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 --2- 172.21.15.106:0/1426794386 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74c00f6510 0x7f74c0108910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 >> 172.21.15.106:0/1426794386 conn(0x7f74c00f1c70 msgr2=0x7f74c00f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 shutdown_connections 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:32.881+0000 7f74c6a68700 1 -- 172.21.15.106:0/1426794386 wait complete. 2024-01-18T07:20:32.883 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:33.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:30.905259+0000 mgr.smithi084.sihbqg (mgr.14162) 34 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:33.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:32.880735+0000 mon.smithi084 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.106:0/1426794386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:32.905622+0000 mgr.smithi084.sihbqg (mgr.14162) 35 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:35.031608+0000 mon.smithi084 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:20:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:35.039076+0000 mon.smithi084 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:20:36.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:36.586 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:34.905927+0000 mgr.smithi084.sihbqg (mgr.14162) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:39.041 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 -- 172.21.15.106:0/4082481044 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 msgr2=0x7fd7440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:39.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 --2- 172.21.15.106:0/4082481044 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7440f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd72c009fb0 tx=0x7fd72c009b30).stop 2024-01-18T07:20:39.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 -- 172.21.15.106:0/4082481044 shutdown_connections 2024-01-18T07:20:39.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 --2- 172.21.15.106:0/4082481044 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7440f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:39.042 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 -- 172.21.15.106:0/4082481044 >> 172.21.15.106:0/4082481044 conn(0x7fd7440f0b70 msgr2=0x7fd7440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 -- 172.21.15.106:0/4082481044 shutdown_connections 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.036+0000 7fd74a715700 1 -- 172.21.15.106:0/4082481044 wait complete. 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd74a715700 1 Processor -- start 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd74a715700 1 -- start start 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd74a715700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:39.043 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd74a715700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd744060280 con 0x7fd7440f5410 2024-01-18T07:20:39.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd743fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:39.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd743fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:45680/0 (socket says 172.21.15.106:45680) 2024-01-18T07:20:39.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.037+0000 7fd743fff700 1 -- 172.21.15.106:0/633673199 learned_addr learned my addr 172.21.15.106:0/633673199 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:39.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd743fff700 1 -- 172.21.15.106:0/633673199 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c005040 con 0x7fd7440f5410 2024-01-18T07:20:39.044 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd743fff700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd72c013510 tx=0x7fd72c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd72c009320 con 0x7fd7440f5410 2024-01-18T07:20:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd744060480 con 0x7fd7440f5410 2024-01-18T07:20:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd72c013f00 con 0x7fd7440f5410 2024-01-18T07:20:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd72c014d90 con 0x7fd7440f5410 2024-01-18T07:20:39.045 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.038+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd744109b20 con 0x7fd7440f5410 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.039+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd72c0095f0 con 0x7fd7440f5410 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.039+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd744045ad0 con 0x7fd7440f5410 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.039+0000 7fd740ff9700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 0x7fd730032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.039+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd72c040f80 con 0x7fd7440f5410 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.039+0000 7fd7437fe700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 0x7fd730032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:39.046 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.040+0000 7fd7437fe700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 0x7fd730032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd7340096a0 tx=0x7fd734006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:39.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.044+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd72c014420 con 0x7fd7440f5410 2024-01-18T07:20:39.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.247+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd74410a340 con 0x7fd7440f5410 2024-01-18T07:20:39.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.247+0000 7fd740ff9700 1 -- 172.21.15.106:0/633673199 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd72c01d7f0 con 0x7fd7440f5410 2024-01-18T07:20:39.249 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:39.249 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 msgr2=0x7fd730032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 0x7fd730032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fd7340096a0 tx=0x7fd734006b40).stop 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 msgr2=0x7fd7441093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fd72c013510 tx=0x7fd72c00b040).stop 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 shutdown_connections 2024-01-18T07:20:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd73002ffb0 0x7fd730032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 --2- 172.21.15.106:0/633673199 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd7440f5410 0x7fd7441093f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 >> 172.21.15.106:0/633673199 conn(0x7fd7440f0b70 msgr2=0x7fd7440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 shutdown_connections 2024-01-18T07:20:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:39.249+0000 7fd74a715700 1 -- 172.21.15.106:0/633673199 wait complete. 2024-01-18T07:20:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:36.906387+0000 mgr.smithi084.sihbqg (mgr.14162) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:39.248880+0000 mon.smithi084 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.106:0/633673199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:40.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:40.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:38.906703+0000 mgr.smithi084.sihbqg (mgr.14162) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 -- 172.21.15.106:0/4728914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 msgr2=0x7f28940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 --2- 172.21.15.106:0/4728914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f28940f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f2884009fb0 tx=0x7f2884009b30).stop 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 -- 172.21.15.106:0/4728914 shutdown_connections 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 --2- 172.21.15.106:0/4728914 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f28940f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 -- 172.21.15.106:0/4728914 >> 172.21.15.106:0/4728914 conn(0x7f28940f0b70 msgr2=0x7f28940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 -- 172.21.15.106:0/4728914 shutdown_connections 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.734+0000 7f289a880700 1 -- 172.21.15.106:0/4728914 wait complete. 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f289a880700 1 Processor -- start 2024-01-18T07:20:41.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f289a880700 1 -- start start 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f289a880700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f289a880700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2894060220 con 0x7f28940f5410 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f2893fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f2893fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:45696/0 (socket says 172.21.15.106:45696) 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.735+0000 7f2893fff700 1 -- 172.21.15.106:0/3549435272 learned_addr learned my addr 172.21.15.106:0/3549435272 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f2893fff700 1 -- 172.21.15.106:0/3549435272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2884005040 con 0x7f28940f5410 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f2893fff700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2884013510 tx=0x7f288400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2884009320 con 0x7f28940f5410 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2894060420 con 0x7f28940f5410 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2884013f00 con 0x7f28940f5410 2024-01-18T07:20:41.738 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2884014d90 con 0x7f28940f5410 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.736+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894109b00 con 0x7f28940f5410 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28840263a0 con 0x7f28940f5410 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f2890ff9700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 0x7f287c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2884041000 con 0x7f28940f5410 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2894059100 con 0x7f28940f5410 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f28937fe700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 0x7f287c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:41.739 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.737+0000 7f28937fe700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 0x7f287c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f28880096a0 tx=0x7f2888006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:41.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.741+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2884012920 con 0x7f28940f5410 2024-01-18T07:20:41.950 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.949+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f289410a250 con 0x7f28940f5410 2024-01-18T07:20:41.951 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.950+0000 7f2890ff9700 1 -- 172.21.15.106:0/3549435272 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f288402b020 con 0x7f28940f5410 2024-01-18T07:20:41.951 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:41.951 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 msgr2=0x7f287c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 0x7f287c032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f28880096a0 tx=0x7f2888006b40).stop 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 msgr2=0x7f2894109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f2884013510 tx=0x7f288400b040).stop 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 shutdown_connections 2024-01-18T07:20:41.953 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f287c02ff60 0x7f287c032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 --2- 172.21.15.106:0/3549435272 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28940f5410 0x7f2894109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 >> 172.21.15.106:0/3549435272 conn(0x7f28940f0b70 msgr2=0x7f28940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 shutdown_connections 2024-01-18T07:20:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:41.952+0000 7f289a880700 1 -- 172.21.15.106:0/3549435272 wait complete. 2024-01-18T07:20:41.954 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:40.906997+0000 mgr.smithi084.sihbqg (mgr.14162) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:43.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:43.344 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:41.951788+0000 mon.smithi084 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.106:0/3549435272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:42.907363+0000 mgr.smithi084.sihbqg (mgr.14162) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:44.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/4158452229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 msgr2=0x7f2a140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/4158452229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a140f90f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f2a04009fb0 tx=0x7f2a04009b30).stop 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/4158452229 shutdown_connections 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/4158452229 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a140f90f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/4158452229 >> 172.21.15.106:0/4158452229 conn(0x7f2a140f0b70 msgr2=0x7f2a140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/4158452229 shutdown_connections 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/4158452229 wait complete. 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.503+0000 7f2a1a0dd700 1 Processor -- start 2024-01-18T07:20:44.505 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a1a0dd700 1 -- start start 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a1a0dd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a1a0dd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a14106bd0 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a137fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a137fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:45712/0 (socket says 172.21.15.106:45712) 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a137fe700 1 -- 172.21.15.106:0/79374963 learned_addr learned my addr 172.21.15.106:0/79374963 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a137fe700 1 -- 172.21.15.106:0/79374963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04005040 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a137fe700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2a04013510 tx=0x7f2a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a04009320 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a14106dd0 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.504+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a04013f00 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a04014dc0 con 0x7f2a14060280 2024-01-18T07:20:44.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a14107260 con 0x7f2a14060280 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a0401d3d0 con 0x7f2a14060280 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a0bfff700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 0x7f2a00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a040410d0 con 0x7f2a14060280 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.505+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a14045ad0 con 0x7f2a14060280 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.506+0000 7f2a12ffd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 0x7f2a00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:44.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.506+0000 7f2a12ffd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 0x7f2a00032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f29fc0096a0 tx=0x7f29fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:44.511 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.510+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a04012920 con 0x7f2a14060280 2024-01-18T07:20:44.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.715+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a140fbf50 con 0x7f2a14060280 2024-01-18T07:20:44.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.716+0000 7f2a0bfff700 1 -- 172.21.15.106:0/79374963 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2a04012330 con 0x7f2a14060280 2024-01-18T07:20:44.717 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:44.717 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 msgr2=0x7f2a00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 0x7f2a00032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f29fc0096a0 tx=0x7f29fc006b40).stop 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 msgr2=0x7f2a141066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f2a04013510 tx=0x7f2a0400b040).stop 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 shutdown_connections 2024-01-18T07:20:44.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a0002ff60 0x7f2a00032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:44.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 --2- 172.21.15.106:0/79374963 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a14060280 0x7f2a141066e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:44.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 >> 172.21.15.106:0/79374963 conn(0x7f2a140f0b70 msgr2=0x7f2a140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:44.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 shutdown_connections 2024-01-18T07:20:44.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:44.718+0000 7f2a1a0dd700 1 -- 172.21.15.106:0/79374963 wait complete. 2024-01-18T07:20:44.720 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:44.717828+0000 mon.smithi084 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.106:0/79374963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:46.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:46.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:47.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:44.907632+0000 mgr.smithi084.sihbqg (mgr.14162) 41 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 -- 172.21.15.106:0/3129018250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 msgr2=0x7fb3a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 --2- 172.21.15.106:0/3129018250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a40f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb398009fb0 tx=0x7fb398009b30).stop 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 -- 172.21.15.106:0/3129018250 shutdown_connections 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 --2- 172.21.15.106:0/3129018250 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a40f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 -- 172.21.15.106:0/3129018250 >> 172.21.15.106:0/3129018250 conn(0x7fb3a40f1c70 msgr2=0x7fb3a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:47.285 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 -- 172.21.15.106:0/3129018250 shutdown_connections 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 -- 172.21.15.106:0/3129018250 wait complete. 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.283+0000 7fb3a9819700 1 Processor -- start 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a9819700 1 -- start start 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a9819700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a9819700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a4106c20 con 0x7fb3a40f6510 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:49658/0 (socket says 172.21.15.106:49658) 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a2ffd700 1 -- 172.21.15.106:0/1153387464 learned_addr learned my addr 172.21.15.106:0/1153387464 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a2ffd700 1 -- 172.21.15.106:0/1153387464 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398005040 con 0x7fb3a40f6510 2024-01-18T07:20:47.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a2ffd700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb398013510 tx=0x7fb39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:47.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb398009320 con 0x7fb3a40f6510 2024-01-18T07:20:47.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a4106e20 con 0x7fb3a40f6510 2024-01-18T07:20:47.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb398013f00 con 0x7fb3a40f6510 2024-01-18T07:20:47.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.284+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb398014d90 con 0x7fb3a40f6510 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.285+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a41072b0 con 0x7fb3a40f6510 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.285+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3980263a0 con 0x7fb3a40f6510 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.285+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4045ad0 con 0x7fb3a40f6510 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.285+0000 7fb38b7fe700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.285+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb398040ff0 con 0x7fb3a40f6510 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.286+0000 7fb3a27fc700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:47.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.286+0000 7fb3a27fc700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 0x7fb38c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb3940096a0 tx=0x7fb394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:47.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.290+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3980095f0 con 0x7fb3a40f6510 2024-01-18T07:20:47.495 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.494+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3a40fbf60 con 0x7fb3a40f6510 2024-01-18T07:20:47.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.495+0000 7fb38b7fe700 1 -- 172.21.15.106:0/1153387464 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb39801db70 con 0x7fb3a40f6510 2024-01-18T07:20:47.496 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:47.496 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 msgr2=0x7fb38c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 0x7fb38c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb3940096a0 tx=0x7fb394006b40).stop 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 msgr2=0x7fb3a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb398013510 tx=0x7fb39800b040).stop 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 shutdown_connections 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb38c02ff60 0x7fb38c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 --2- 172.21.15.106:0/1153387464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb3a40f6510 0x7fb3a4106730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 >> 172.21.15.106:0/1153387464 conn(0x7fb3a40f1c70 msgr2=0x7fb3a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:47.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 shutdown_connections 2024-01-18T07:20:47.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:47.497+0000 7fb3a9819700 1 -- 172.21.15.106:0/1153387464 wait complete. 2024-01-18T07:20:47.499 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:47.496324+0000 mon.smithi084 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.106:0/1153387464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:48.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:48.875 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:49.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:46.907919+0000 mgr.smithi084.sihbqg (mgr.14162) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 -- 172.21.15.106:0/1230235494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 msgr2=0x7f255c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 --2- 172.21.15.106:0/1230235494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c0f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2544009fb0 tx=0x7f2544009b30).stop 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 -- 172.21.15.106:0/1230235494 shutdown_connections 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 --2- 172.21.15.106:0/1230235494 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c0f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 -- 172.21.15.106:0/1230235494 >> 172.21.15.106:0/1230235494 conn(0x7f255c0f0b70 msgr2=0x7f255c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 -- 172.21.15.106:0/1230235494 shutdown_connections 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.028+0000 7f2560a94700 1 -- 172.21.15.106:0/1230235494 wait complete. 2024-01-18T07:20:50.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f2560a94700 1 Processor -- start 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f2560a94700 1 -- start start 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f2560a94700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f2560a94700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c105b20 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f255a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f255a59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:49674/0 (socket says 172.21.15.106:49674) 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.029+0000 7f255a59c700 1 -- 172.21.15.106:0/566039434 learned_addr learned my addr 172.21.15.106:0/566039434 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f255a59c700 1 -- 172.21.15.106:0/566039434 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544005040 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f255a59c700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2544013510 tx=0x7f254400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2544009320 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255c105d20 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2544013f00 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2544014d90 con 0x7f255c0f5410 2024-01-18T07:20:50.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.030+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255c1061b0 con 0x7f255c0f5410 2024-01-18T07:20:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25440095f0 con 0x7f255c0f5410 2024-01-18T07:20:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255c045ad0 con 0x7f255c0f5410 2024-01-18T07:20:50.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2552ffd700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 0x7f2548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2544040fe0 con 0x7f255c0f5410 2024-01-18T07:20:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2559d9b700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 0x7f2548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:50.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.031+0000 7f2559d9b700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 0x7f2548032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f254c0096a0 tx=0x7f254c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:50.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.035+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f254401d7f0 con 0x7f255c0f5410 2024-01-18T07:20:50.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.241+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f255c0fae60 con 0x7f255c0f5410 2024-01-18T07:20:50.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.242+0000 7f2552ffd700 1 -- 172.21.15.106:0/566039434 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2544010230 con 0x7f255c0f5410 2024-01-18T07:20:50.243 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:50.243 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:50.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 msgr2=0x7f2548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:50.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 0x7f2548032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f254c0096a0 tx=0x7f254c006b40).stop 2024-01-18T07:20:50.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 msgr2=0x7f255c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2544013510 tx=0x7f254400b040).stop 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 shutdown_connections 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f254802ff60 0x7f2548032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 --2- 172.21.15.106:0/566039434 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f255c0f5410 0x7f255c105630 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 >> 172.21.15.106:0/566039434 conn(0x7f255c0f0b70 msgr2=0x7f255c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 shutdown_connections 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:50.244+0000 7f2560a94700 1 -- 172.21.15.106:0/566039434 wait complete. 2024-01-18T07:20:50.246 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:51.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:48.908199+0000 mgr.smithi084.sihbqg (mgr.14162) 43 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:51.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:50.243686+0000 mon.smithi084 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.106:0/566039434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:51.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:51.625 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 -- 172.21.15.106:0/1851414815 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 msgr2=0x7f6ee00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 --2- 172.21.15.106:0/1851414815 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee00f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f6ec8009fb0 tx=0x7f6ec8009b30).stop 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 -- 172.21.15.106:0/1851414815 shutdown_connections 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 --2- 172.21.15.106:0/1851414815 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee00f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 -- 172.21.15.106:0/1851414815 >> 172.21.15.106:0/1851414815 conn(0x7f6ee00f0b70 msgr2=0x7f6ee00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:52.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 -- 172.21.15.106:0/1851414815 shutdown_connections 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 -- 172.21.15.106:0/1851414815 wait complete. 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.765+0000 7f6ee658f700 1 Processor -- start 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6ee658f700 1 -- start start 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6ee658f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6ee658f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0106c00 con 0x7f6ee00f5410 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6edffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6edffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:49688/0 (socket says 172.21.15.106:49688) 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6edffff700 1 -- 172.21.15.106:0/2633369712 learned_addr learned my addr 172.21.15.106:0/2633369712 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6edffff700 1 -- 172.21.15.106:0/2633369712 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8005040 con 0x7f6ee00f5410 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.766+0000 7f6edffff700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6ec8014000 tx=0x7f6ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec80095f0 con 0x7f6ee00f5410 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee0106e00 con 0x7f6ee00f5410 2024-01-18T07:20:52.769 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec8007e70 con 0x7f6ee00f5410 2024-01-18T07:20:52.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec8014ea0 con 0x7f6ee00f5410 2024-01-18T07:20:52.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0107290 con 0x7f6ee00f5410 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.767+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ec8025070 con 0x7f6ee00f5410 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.768+0000 7f6edcff9700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 0x7f6ecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.768+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ec8040970 con 0x7f6ee00f5410 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.768+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0059100 con 0x7f6ee00f5410 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.768+0000 7f6edf7fe700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 0x7f6ecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:52.771 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.768+0000 7f6edf7fe700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 0x7f6ecc032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f6ed00096a0 tx=0x7f6ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:52.773 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.773+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ec8026350 con 0x7f6ee00f5410 2024-01-18T07:20:52.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.978+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ee00fae60 con 0x7f6ee00f5410 2024-01-18T07:20:52.980 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.979+0000 7f6edcff9700 1 -- 172.21.15.106:0/2633369712 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ec8012330 con 0x7f6ee00f5410 2024-01-18T07:20:52.981 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:52.981 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:52.981 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 msgr2=0x7f6ecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 0x7f6ecc032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f6ed00096a0 tx=0x7f6ed0006b40).stop 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 msgr2=0x7f6ee0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6ec8014000 tx=0x7f6ec800b040).stop 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 shutdown_connections 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6ecc02ff60 0x7f6ecc032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 --2- 172.21.15.106:0/2633369712 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ee00f5410 0x7f6ee0106710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 >> 172.21.15.106:0/2633369712 conn(0x7f6ee00f0b70 msgr2=0x7f6ee005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 shutdown_connections 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:52.980+0000 7f6ee658f700 1 -- 172.21.15.106:0/2633369712 wait complete. 2024-01-18T07:20:52.982 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:53.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:50.908600+0000 mgr.smithi084.sihbqg (mgr.14162) 44 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:54.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:54.370 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:54.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:52.980110+0000 mon.smithi084 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.106:0/2633369712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:52.908921+0000 mgr.smithi084.sihbqg (mgr.14162) 45 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/364425892 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 msgr2=0x7fcfa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/364425892 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa40f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fcf8c009fb0 tx=0x7fcf8c009b30).stop 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/364425892 shutdown_connections 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/364425892 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa40f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/364425892 >> 172.21.15.106:0/364425892 conn(0x7fcfa40f0b70 msgr2=0x7fcfa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:55.543 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/364425892 shutdown_connections 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.541+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/364425892 wait complete. 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa8ba0700 1 Processor -- start 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa8ba0700 1 -- start start 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa8ba0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa8ba0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4106c00 con 0x7fcfa40f5410 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37320/0 (socket says 172.21.15.106:37320) 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa259c700 1 -- 172.21.15.106:0/2554514059 learned_addr learned my addr 172.21.15.106:0/2554514059 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.542+0000 7fcfa259c700 1 -- 172.21.15.106:0/2554514059 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c005040 con 0x7fcfa40f5410 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcfa259c700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcf8c014000 tx=0x7fcf8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf8c0095f0 con 0x7fcfa40f5410 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf8c007e70 con 0x7fcfa40f5410 2024-01-18T07:20:55.544 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa4106e00 con 0x7fcfa40f5410 2024-01-18T07:20:55.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf8c014ea0 con 0x7fcfa40f5410 2024-01-18T07:20:55.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa4107290 con 0x7fcfa40f5410 2024-01-18T07:20:55.545 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.543+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf8c025070 con 0x7fcfa40f5410 2024-01-18T07:20:55.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.544+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa4045ad0 con 0x7fcfa40f5410 2024-01-18T07:20:55.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.544+0000 7fcf9affd700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:55.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.544+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf8c040970 con 0x7fcfa40f5410 2024-01-18T07:20:55.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.544+0000 7fcfa1d9b700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:55.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.544+0000 7fcfa1d9b700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fcf940096a0 tx=0x7fcf94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:55.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.548+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf8c0079c0 con 0x7fcfa40f5410 2024-01-18T07:20:55.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.756+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfa4108140 con 0x7fcfa40f5410 2024-01-18T07:20:55.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.756+0000 7fcf9affd700 1 -- 172.21.15.106:0/2554514059 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf8c00f300 con 0x7fcfa40f5410 2024-01-18T07:20:55.757 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:55.757 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:55.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 msgr2=0x7fcf90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:55.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 0x7fcf90032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fcf940096a0 tx=0x7fcf94006b40).stop 2024-01-18T07:20:55.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 msgr2=0x7fcfa4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:55.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcf8c014000 tx=0x7fcf8c00b040).stop 2024-01-18T07:20:55.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 shutdown_connections 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcf9002ff60 0x7fcf90032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 --2- 172.21.15.106:0/2554514059 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcfa40f5410 0x7fcfa4106710 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 >> 172.21.15.106:0/2554514059 conn(0x7fcfa40f0b70 msgr2=0x7fcfa405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 shutdown_connections 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:55.758+0000 7fcfa8ba0700 1 -- 172.21.15.106:0/2554514059 wait complete. 2024-01-18T07:20:55.759 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:56.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:55.757064+0000 mon.smithi084 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.106:0/2554514059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:57.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:57.133 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:20:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:54.909156+0000 mgr.smithi084.sihbqg (mgr.14162) 46 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:58.289 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 -- 172.21.15.106:0/179064412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 msgr2=0x7f9ad40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 --2- 172.21.15.106:0/179064412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad40f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f9ac4009fb0 tx=0x7f9ac4009b30).stop 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 -- 172.21.15.106:0/179064412 shutdown_connections 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 --2- 172.21.15.106:0/179064412 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad40f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 -- 172.21.15.106:0/179064412 >> 172.21.15.106:0/179064412 conn(0x7f9ad40f0b70 msgr2=0x7f9ad40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 -- 172.21.15.106:0/179064412 shutdown_connections 2024-01-18T07:20:58.290 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.289+0000 7f9adb519700 1 -- 172.21.15.106:0/179064412 wait complete. 2024-01-18T07:20:58.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.290+0000 7f9adb519700 1 Processor -- start 2024-01-18T07:20:58.291 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.290+0000 7f9adb519700 1 -- start start 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.290+0000 7f9adb519700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.290+0000 7f9adb519700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad4105b20 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.290+0000 7f9ad92b5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ad92b5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37340/0 (socket says 172.21.15.106:37340) 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ad92b5700 1 -- 172.21.15.106:0/373877757 learned_addr learned my addr 172.21.15.106:0/373877757 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ad92b5700 1 -- 172.21.15.106:0/373877757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac4005040 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ad92b5700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9ac4013510 tx=0x7f9ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac4009320 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad4105d20 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ac4013f00 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ac4014d90 con 0x7f9ad40f5410 2024-01-18T07:20:58.292 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.291+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41061b0 con 0x7f9ad40f5410 2024-01-18T07:20:58.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.292+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ac40263a0 con 0x7f9ad40f5410 2024-01-18T07:20:58.293 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.292+0000 7f9ac9ffb700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 0x7f9ac0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:20:58.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.292+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ac40410b0 con 0x7f9ad40f5410 2024-01-18T07:20:58.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.292+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad4045ad0 con 0x7f9ad40f5410 2024-01-18T07:20:58.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.293+0000 7f9ad8ab4700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 0x7f9ac0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:20:58.294 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.293+0000 7f9ad8ab4700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 0x7f9ac0032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:20:58.297 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.297+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ac4012920 con 0x7f9ad40f5410 2024-01-18T07:20:58.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.503+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ad40fae60 con 0x7f9ad40f5410 2024-01-18T07:20:58.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.504+0000 7f9ac9ffb700 1 -- 172.21.15.106:0/373877757 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ac402b020 con 0x7f9ad40f5410 2024-01-18T07:20:58.504 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:20:58.504 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 msgr2=0x7f9ac0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 0x7f9ac0032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9ad00096a0 tx=0x7f9ad0006b40).stop 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 msgr2=0x7f9ad4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9ac4013510 tx=0x7f9ac400b040).stop 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 shutdown_connections 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ac002ffb0 0x7f9ac0032470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 --2- 172.21.15.106:0/373877757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ad40f5410 0x7f9ad4105630 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 >> 172.21.15.106:0/373877757 conn(0x7f9ad40f0b70 msgr2=0x7f9ad40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 shutdown_connections 2024-01-18T07:20:58.506 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:20:58.505+0000 7f9adb519700 1 -- 172.21.15.106:0/373877757 wait complete. 2024-01-18T07:20:58.507 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:20:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:56.909455+0000 mgr.smithi084.sihbqg (mgr.14162) 47 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:20:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:20:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:20:58.504452+0000 mon.smithi084 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.106:0/373877757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:20:59.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:20:59.882 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:01.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 -- 172.21.15.106:0/2730460274 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 msgr2=0x7f2ab80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:01.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/2730460274 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab80f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f2aa8009fb0 tx=0x7f2aa8009b30).stop 2024-01-18T07:21:01.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 -- 172.21.15.106:0/2730460274 shutdown_connections 2024-01-18T07:21:01.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/2730460274 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab80f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:01.059 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 -- 172.21.15.106:0/2730460274 >> 172.21.15.106:0/2730460274 conn(0x7f2ab80f1c70 msgr2=0x7f2ab80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 -- 172.21.15.106:0/2730460274 shutdown_connections 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.057+0000 7f2abf8b9700 1 -- 172.21.15.106:0/2730460274 wait complete. 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abf8b9700 1 Processor -- start 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abf8b9700 1 -- start start 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abf8b9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abf8b9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8069e20 con 0x7f2ab80f6510 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abd655700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abd655700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37352/0 (socket says 172.21.15.106:37352) 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.058+0000 7f2abd655700 1 -- 172.21.15.106:0/1069520299 learned_addr learned my addr 172.21.15.106:0/1069520299 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2abd655700 1 -- 172.21.15.106:0/1069520299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8005040 con 0x7f2ab80f6510 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2abd655700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2aa8013510 tx=0x7f2aa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa8009320 con 0x7f2ab80f6510 2024-01-18T07:21:01.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab8069fc0 con 0x7f2ab80f6510 2024-01-18T07:21:01.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2aa8013f00 con 0x7f2ab80f6510 2024-01-18T07:21:01.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2aa8014d90 con 0x7f2ab80f6510 2024-01-18T07:21:01.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.059+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab806a450 con 0x7f2ab80f6510 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.060+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2aa80095f0 con 0x7f2ab80f6510 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.060+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab8059100 con 0x7f2ab80f6510 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.060+0000 7f2aadffb700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.060+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2aa8040fc0 con 0x7f2ab80f6510 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.060+0000 7f2abce54700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:01.062 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.061+0000 7f2abce54700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 0x7f2aa4032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f2ab40096a0 tx=0x7f2ab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:01.065 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.065+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2aa8014420 con 0x7f2ab80f6510 2024-01-18T07:21:01.275 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.272+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ab80fbf60 con 0x7f2ab80f6510 2024-01-18T07:21:01.275 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.273+0000 7f2aadffb700 1 -- 172.21.15.106:0/1069520299 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2aa8010230 con 0x7f2ab80f6510 2024-01-18T07:21:01.276 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:01.276 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 msgr2=0x7f2aa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 0x7f2aa4032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f2ab40096a0 tx=0x7f2ab4006b40).stop 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 msgr2=0x7f2ab806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2aa8013510 tx=0x7f2aa800b040).stop 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 shutdown_connections 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2aa402ff60 0x7f2aa4032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 --2- 172.21.15.106:0/1069520299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2ab80f6510 0x7f2ab806b9a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 >> 172.21.15.106:0/1069520299 conn(0x7f2ab80f1c70 msgr2=0x7f2ab80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 shutdown_connections 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:01.275+0000 7f2abf8b9700 1 -- 172.21.15.106:0/1069520299 wait complete. 2024-01-18T07:21:01.277 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:20:58.909741+0000 mgr.smithi084.sihbqg (mgr.14162) 48 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:01.273467+0000 mon.smithi084 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.106:0/1069520299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:02.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:02.664 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:00.910044+0000 mgr.smithi084.sihbqg (mgr.14162) 49 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 -- 172.21.15.106:0/2484983166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 msgr2=0x7f03200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 --2- 172.21.15.106:0/2484983166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f03200f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0310009fb0 tx=0x7f0310009b30).stop 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 -- 172.21.15.106:0/2484983166 shutdown_connections 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 --2- 172.21.15.106:0/2484983166 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f03200f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 -- 172.21.15.106:0/2484983166 >> 172.21.15.106:0/2484983166 conn(0x7f03200f0b70 msgr2=0x7f03200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 -- 172.21.15.106:0/2484983166 shutdown_connections 2024-01-18T07:21:03.794 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.793+0000 7f0324c3e700 1 -- 172.21.15.106:0/2484983166 wait complete. 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.794+0000 7f0324c3e700 1 Processor -- start 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.794+0000 7f0324c3e700 1 -- start start 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.794+0000 7f0324c3e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.794+0000 7f0324c3e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0320105b20 con 0x7f03200f5410 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f031e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f031e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37378/0 (socket says 172.21.15.106:37378) 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f031e59c700 1 -- 172.21.15.106:0/560561538 learned_addr learned my addr 172.21.15.106:0/560561538 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f031e59c700 1 -- 172.21.15.106:0/560561538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310005040 con 0x7f03200f5410 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f031e59c700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0310014000 tx=0x7f031000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:03.796 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03100095f0 con 0x7f03200f5410 2024-01-18T07:21:03.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0320105d20 con 0x7f03200f5410 2024-01-18T07:21:03.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0310007e70 con 0x7f03200f5410 2024-01-18T07:21:03.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.795+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0310014ea0 con 0x7f03200f5410 2024-01-18T07:21:03.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.796+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03201061b0 con 0x7f03200f5410 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.796+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0310014690 con 0x7f03200f5410 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.796+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0320045ad0 con 0x7f03200f5410 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.797+0000 7f030effd700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 0x7f0308032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.797+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0310040830 con 0x7f03200f5410 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.797+0000 7f031dd9b700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 0x7f0308032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:03.798 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.797+0000 7f031dd9b700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 0x7f0308032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f03140096a0 tx=0x7f0314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:03.801 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:03.801+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f031000f300 con 0x7f03200f5410 2024-01-18T07:21:04.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.011+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03200fae60 con 0x7f03200f5410 2024-01-18T07:21:04.012 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.012+0000 7f030effd700 1 -- 172.21.15.106:0/560561538 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f031000f300 con 0x7f03200f5410 2024-01-18T07:21:04.012 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:04.013 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 msgr2=0x7f0308032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 0x7f0308032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f03140096a0 tx=0x7f0314006b40).stop 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 msgr2=0x7f0320105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0310014000 tx=0x7f031000b040).stop 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 shutdown_connections 2024-01-18T07:21:04.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f030802ffb0 0x7f0308032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:04.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 --2- 172.21.15.106:0/560561538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f03200f5410 0x7f0320105630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:04.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 >> 172.21.15.106:0/560561538 conn(0x7f03200f0b70 msgr2=0x7f03200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:04.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 shutdown_connections 2024-01-18T07:21:04.015 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:04.014+0000 7f0324c3e700 1 -- 172.21.15.106:0/560561538 wait complete. 2024-01-18T07:21:04.015 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:04.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:02.910340+0000 mgr.smithi084.sihbqg (mgr.14162) 50 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:04.012655+0000 mon.smithi084 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.106:0/560561538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:05.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:05.659 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.881+0000 7ff8a200a700 1 -- 172.21.15.106:0/4093015449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 msgr2=0x7ff89c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.881+0000 7ff8a200a700 1 --2- 172.21.15.106:0/4093015449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c0f66c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff88c009fb0 tx=0x7ff88c009b30).stop 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.881+0000 7ff8a200a700 1 -- 172.21.15.106:0/4093015449 shutdown_connections 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.881+0000 7ff8a200a700 1 --2- 172.21.15.106:0/4093015449 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c0f66c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.881+0000 7ff8a200a700 1 -- 172.21.15.106:0/4093015449 >> 172.21.15.106:0/4093015449 conn(0x7ff89c0f1aa0 msgr2=0x7ff89c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 -- 172.21.15.106:0/4093015449 shutdown_connections 2024-01-18T07:21:06.882 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 -- 172.21.15.106:0/4093015449 wait complete. 2024-01-18T07:21:06.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 Processor -- start 2024-01-18T07:21:06.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 -- start start 2024-01-18T07:21:06.884 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.882+0000 7ff8a200a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c1069a0 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a1008700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a1008700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55442/0 (socket says 172.21.15.106:55442) 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a1008700 1 -- 172.21.15.106:0/50744447 learned_addr learned my addr 172.21.15.106:0/50744447 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a1008700 1 -- 172.21.15.106:0/50744447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c005040 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a1008700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff88c013510 tx=0x7ff88c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff88c009320 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff88c013f00 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c106ba0 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff88c014dc0 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.883+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c107030 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.884+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff88c0095f0 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.884+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89c045ad0 con 0x7ff89c0f62a0 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.884+0000 7ff891ffb700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 0x7ff888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:06.885 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.884+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff88c041020 con 0x7ff89c0f62a0 2024-01-18T07:21:06.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.884+0000 7ff8a0807700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 0x7ff888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:06.886 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.885+0000 7ff8a0807700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 0x7ff888032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff8980096a0 tx=0x7ff898006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:06.889 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:06.889+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff88c014420 con 0x7ff89c0f62a0 2024-01-18T07:21:07.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.095+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff89c107e60 con 0x7ff89c0f62a0 2024-01-18T07:21:07.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.096+0000 7ff891ffb700 1 -- 172.21.15.106:0/50744447 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff88c01c920 con 0x7ff89c0f62a0 2024-01-18T07:21:07.096 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:07.097 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:07.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 msgr2=0x7ff888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:07.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 0x7ff888032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff8980096a0 tx=0x7ff898006b40).stop 2024-01-18T07:21:07.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 msgr2=0x7ff89c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7ff88c013510 tx=0x7ff88c00b040).stop 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 shutdown_connections 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff88802ff60 0x7ff888032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 --2- 172.21.15.106:0/50744447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff89c0f62a0 0x7ff89c1064b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 >> 172.21.15.106:0/50744447 conn(0x7ff89c0f1aa0 msgr2=0x7ff89c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 shutdown_connections 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:07.098+0000 7ff8a200a700 1 -- 172.21.15.106:0/50744447 wait complete. 2024-01-18T07:21:07.099 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:04.910637+0000 mgr.smithi084.sihbqg (mgr.14162) 51 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:08.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:07.096622+0000 mon.smithi084 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.106:0/50744447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:08.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:08.525 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:09.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:06.910997+0000 mgr.smithi084.sihbqg (mgr.14162) 52 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.750+0000 7f59ddcd0700 1 -- 172.21.15.106:0/3353260001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 msgr2=0x7f59d80f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.750+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/3353260001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f8f80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f59cc009fb0 tx=0x7f59cc009b30).stop 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 -- 172.21.15.106:0/3353260001 shutdown_connections 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/3353260001 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f8f80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 -- 172.21.15.106:0/3353260001 >> 172.21.15.106:0/3353260001 conn(0x7f59d80f09a0 msgr2=0x7f59d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 -- 172.21.15.106:0/3353260001 shutdown_connections 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 -- 172.21.15.106:0/3353260001 wait complete. 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 Processor -- start 2024-01-18T07:21:09.753 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.751+0000 7f59ddcd0700 1 -- start start 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59ddcd0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59ddcd0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80f4e80 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59dccce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59dccce700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55454/0 (socket says 172.21.15.106:55454) 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59dccce700 1 -- 172.21.15.106:0/4183538235 learned_addr learned my addr 172.21.15.106:0/4183538235 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59dccce700 1 -- 172.21.15.106:0/4183538235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc005040 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59dccce700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f59cc013510 tx=0x7f59cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59cc009320 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d80f5020 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59cc013f00 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59cc014d90 con 0x7f59d8060180 2024-01-18T07:21:09.754 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.752+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d80f54b0 con 0x7f59d8060180 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.753+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59cc014420 con 0x7f59d8060180 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.753+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8059100 con 0x7f59d8060180 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.753+0000 7f59d57fa700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 0x7f59c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.753+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59cc041000 con 0x7f59d8060180 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.754+0000 7f59d7fff700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 0x7f59c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:09.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.754+0000 7f59d7fff700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 0x7f59c8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:09.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.758+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59cc0095f0 con 0x7f59d8060180 2024-01-18T07:21:09.963 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.963+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59d80fbce0 con 0x7f59d8060180 2024-01-18T07:21:09.964 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.963+0000 7f59d57fa700 1 -- 172.21.15.106:0/4183538235 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59cc01db70 con 0x7f59d8060180 2024-01-18T07:21:09.964 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:09.964 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 msgr2=0x7f59c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 0x7f59c8032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).stop 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 msgr2=0x7f59d80f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f59cc013510 tx=0x7f59cc00b040).stop 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 shutdown_connections 2024-01-18T07:21:09.965 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f59c802ff60 0x7f59c8032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:09.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 --2- 172.21.15.106:0/4183538235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f59d8060180 0x7f59d80f6a00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:09.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 >> 172.21.15.106:0/4183538235 conn(0x7f59d80f09a0 msgr2=0x7f59d80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:09.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 shutdown_connections 2024-01-18T07:21:09.966 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:09.965+0000 7f59ddcd0700 1 -- 172.21.15.106:0/4183538235 wait complete. 2024-01-18T07:21:09.966 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:11.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:11.362 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:08.911242+0000 mgr.smithi084.sihbqg (mgr.14162) 53 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:09.964195+0000 mon.smithi084 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.106:0/4183538235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:12.553 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.551+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1553371428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 msgr2=0x7f4a9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.551+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1553371428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4a8c009fb0 tx=0x7f4a8c009b30).stop 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.551+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1553371428 shutdown_connections 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.551+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1553371428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.551+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1553371428 >> 172.21.15.106:0/1553371428 conn(0x7f4a9c0f0b70 msgr2=0x7f4a9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1553371428 shutdown_connections 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1553371428 wait complete. 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 Processor -- start 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 -- start start 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa45e6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c0fb820 con 0x7f4a9c0f5410 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa2382700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa2382700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55476/0 (socket says 172.21.15.106:55476) 2024-01-18T07:21:12.554 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.552+0000 7f4aa2382700 1 -- 172.21.15.106:0/1212194923 learned_addr learned my addr 172.21.15.106:0/1212194923 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4aa2382700 1 -- 172.21.15.106:0/1212194923 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c005040 con 0x7f4a9c0f5410 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4aa2382700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4a8c013510 tx=0x7f4a8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8c009320 con 0x7f4a9c0f5410 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c0fb9c0 con 0x7f4a9c0f5410 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a8c013f00 con 0x7f4a9c0f5410 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a8c014d90 con 0x7f4a9c0f5410 2024-01-18T07:21:12.555 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.553+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c0fbe50 con 0x7f4a9c0f5410 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.554+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c059100 con 0x7f4a9c0f5410 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.554+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a8c0095f0 con 0x7f4a9c0f5410 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.554+0000 7f4a92ffd700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 0x7f4a88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.554+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a8c0410c0 con 0x7f4a9c0f5410 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.554+0000 7f4aa1b81700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 0x7f4a88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:12.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.555+0000 7f4aa1b81700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 0x7f4a88032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a980096a0 tx=0x7f4a98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:12.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.559+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a8c014420 con 0x7f4a9c0f5410 2024-01-18T07:21:12.765 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.765+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a9c0fae60 con 0x7f4a9c0f5410 2024-01-18T07:21:12.766 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.766+0000 7f4a92ffd700 1 -- 172.21.15.106:0/1212194923 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a8c010230 con 0x7f4a9c0f5410 2024-01-18T07:21:12.766 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:12.766 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 msgr2=0x7f4a88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 0x7f4a88032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a980096a0 tx=0x7f4a98006b40).stop 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 msgr2=0x7f4a9c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4a8c013510 tx=0x7f4a8c00b040).stop 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 shutdown_connections 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a8802ff60 0x7f4a88032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.767+0000 7f4aa45e6700 1 --2- 172.21.15.106:0/1212194923 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a9c0f5410 0x7f4a9c0fd3a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.768+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 >> 172.21.15.106:0/1212194923 conn(0x7f4a9c0f0b70 msgr2=0x7f4a9c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.768+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 shutdown_connections 2024-01-18T07:21:12.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:12.768+0000 7f4aa45e6700 1 -- 172.21.15.106:0/1212194923 wait complete. 2024-01-18T07:21:12.769 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:10.911585+0000 mgr.smithi084.sihbqg (mgr.14162) 54 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:12.766526+0000 mon.smithi084 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.106:0/1212194923' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:14.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:14.156 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.320+0000 7fc6ec315700 1 -- 172.21.15.106:0/2952780169 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 msgr2=0x7fc6e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.320+0000 7fc6ec315700 1 --2- 172.21.15.106:0/2952780169 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e40f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc6d4009fb0 tx=0x7fc6d4009b30).stop 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- 172.21.15.106:0/2952780169 shutdown_connections 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 --2- 172.21.15.106:0/2952780169 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e40f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- 172.21.15.106:0/2952780169 >> 172.21.15.106:0/2952780169 conn(0x7fc6e40f0b70 msgr2=0x7fc6e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- 172.21.15.106:0/2952780169 shutdown_connections 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- 172.21.15.106:0/2952780169 wait complete. 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 Processor -- start 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- start start 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:15.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.321+0000 7fc6ec315700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e4105b20 con 0x7fc6e40f5410 2024-01-18T07:21:15.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ea0b1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:15.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ea0b1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:42232/0 (socket says 172.21.15.106:42232) 2024-01-18T07:21:15.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ea0b1700 1 -- 172.21.15.106:0/3654015801 learned_addr learned my addr 172.21.15.106:0/3654015801 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:15.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ea0b1700 1 -- 172.21.15.106:0/3654015801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d4005040 con 0x7fc6e40f5410 2024-01-18T07:21:15.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ea0b1700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc6d4013510 tx=0x7fc6d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:15.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d4009320 con 0x7fc6e40f5410 2024-01-18T07:21:15.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e4105d20 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6d4013f00 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6d4014d90 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.322+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e41061b0 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.323+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6d40095f0 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.323+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4045ad0 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.323+0000 7fc6daffd700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 0x7fc6d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.323+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6d40410c0 con 0x7fc6e40f5410 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.323+0000 7fc6e98b0700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 0x7fc6d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:15.326 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.324+0000 7fc6e98b0700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 0x7fc6d0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:15.329 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.328+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6d4014420 con 0x7fc6e40f5410 2024-01-18T07:21:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:12.911900+0000 mgr.smithi084.sihbqg (mgr.14162) 55 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:15.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.534+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6e40fae60 con 0x7fc6e40f5410 2024-01-18T07:21:15.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.534+0000 7fc6daffd700 1 -- 172.21.15.106:0/3654015801 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6d4010230 con 0x7fc6e40f5410 2024-01-18T07:21:15.535 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:15.536 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 msgr2=0x7fc6d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 0x7fc6d0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).stop 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 msgr2=0x7fc6e4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fc6d4013510 tx=0x7fc6d400b040).stop 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 shutdown_connections 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc6d002ff60 0x7fc6d0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 --2- 172.21.15.106:0/3654015801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6e40f5410 0x7fc6e4105630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 >> 172.21.15.106:0/3654015801 conn(0x7fc6e40f0b70 msgr2=0x7fc6e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 shutdown_connections 2024-01-18T07:21:15.538 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:15.536+0000 7fc6ec315700 1 -- 172.21.15.106:0/3654015801 wait complete. 2024-01-18T07:21:15.539 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:15.535969+0000 mon.smithi084 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.106:0/3654015801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:16.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:16.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:14.912236+0000 mgr.smithi084.sihbqg (mgr.14162) 56 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1068128054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 msgr2=0x7fc0b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1068128054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b80f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fc0a8009fb0 tx=0x7fc0a8009b30).stop 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1068128054 shutdown_connections 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1068128054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b80f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1068128054 >> 172.21.15.106:0/1068128054 conn(0x7fc0b80f1c70 msgr2=0x7fc0b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1068128054 shutdown_connections 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.090+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1068128054 wait complete. 2024-01-18T07:21:18.092 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0bcde2700 1 Processor -- start 2024-01-18T07:21:18.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0bcde2700 1 -- start start 2024-01-18T07:21:18.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0bcde2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:18.093 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0bcde2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b8106c20 con 0x7fc0b80f6510 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0b659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0b659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:42242/0 (socket says 172.21.15.106:42242) 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.091+0000 7fc0b659c700 1 -- 172.21.15.106:0/1214089890 learned_addr learned my addr 172.21.15.106:0/1214089890 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0b659c700 1 -- 172.21.15.106:0/1214089890 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a8005040 con 0x7fc0b80f6510 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0b659c700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fc0a8013510 tx=0x7fc0a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a8009320 con 0x7fc0b80f6510 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b8106e20 con 0x7fc0b80f6510 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0a8013f00 con 0x7fc0b80f6510 2024-01-18T07:21:18.095 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a8014d90 con 0x7fc0b80f6510 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.092+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b81072b0 con 0x7fc0b80f6510 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0a80263a0 con 0x7fc0b80f6510 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b8045ad0 con 0x7fc0b80f6510 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0a6ffd700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 0x7fc0a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0a80412f0 con 0x7fc0b80f6510 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0b5d9b700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 0x7fc0a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:18.096 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.093+0000 7fc0b5d9b700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 0x7fc0a0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc0ac0096a0 tx=0x7fc0ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:18.098 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.097+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0a8012330 con 0x7fc0b80f6510 2024-01-18T07:21:18.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.305+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0b80fbf60 con 0x7fc0b80f6510 2024-01-18T07:21:18.307 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.305+0000 7fc0a6ffd700 1 -- 172.21.15.106:0/1214089890 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0a8012330 con 0x7fc0b80f6510 2024-01-18T07:21:18.307 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:18.307 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:18.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 msgr2=0x7fc0a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:18.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 0x7fc0a0032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc0ac0096a0 tx=0x7fc0ac006b40).stop 2024-01-18T07:21:18.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 msgr2=0x7fc0b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:18.308 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fc0a8013510 tx=0x7fc0a800b040).stop 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 shutdown_connections 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0a002ff60 0x7fc0a0032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 --2- 172.21.15.106:0/1214089890 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0b80f6510 0x7fc0b8106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 >> 172.21.15.106:0/1214089890 conn(0x7fc0b80f1c70 msgr2=0x7fc0b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 shutdown_connections 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:18.307+0000 7fc0bcde2700 1 -- 172.21.15.106:0/1214089890 wait complete. 2024-01-18T07:21:18.309 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:16.912621+0000 mgr.smithi084.sihbqg (mgr.14162) 57 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:19.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:18.307037+0000 mon.smithi084 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.106:0/1214089890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:19.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:19.701 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:20.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 -- 172.21.15.106:0/3826700402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 msgr2=0x7f16500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:20.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 --2- 172.21.15.106:0/3826700402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f16500f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1640009fb0 tx=0x7f1640009b30).stop 2024-01-18T07:21:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 -- 172.21.15.106:0/3826700402 shutdown_connections 2024-01-18T07:21:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 --2- 172.21.15.106:0/3826700402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f16500f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 -- 172.21.15.106:0/3826700402 >> 172.21.15.106:0/3826700402 conn(0x7f16500f0b70 msgr2=0x7f16500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 -- 172.21.15.106:0/3826700402 shutdown_connections 2024-01-18T07:21:20.894 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.889+0000 7f1657733700 1 -- 172.21.15.106:0/3826700402 wait complete. 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f1657733700 1 Processor -- start 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f1657733700 1 -- start start 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f1657733700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f1657733700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1650106c10 con 0x7f16500f5410 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f16554cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:20.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f16554cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:42254/0 (socket says 172.21.15.106:42254) 2024-01-18T07:21:20.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f16554cf700 1 -- 172.21.15.106:0/58402789 learned_addr learned my addr 172.21.15.106:0/58402789 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:20.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.890+0000 7f16554cf700 1 -- 172.21.15.106:0/58402789 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1640005040 con 0x7f16500f5410 2024-01-18T07:21:20.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.891+0000 7f16554cf700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1640013510 tx=0x7f164000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:20.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640009320 con 0x7f16500f5410 2024-01-18T07:21:20.896 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1650106e10 con 0x7f16500f5410 2024-01-18T07:21:20.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16501072a0 con 0x7f16500f5410 2024-01-18T07:21:20.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1640013f00 con 0x7f16500f5410 2024-01-18T07:21:20.897 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1640014d60 con 0x7f16500f5410 2024-01-18T07:21:20.899 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.892+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16400263a0 con 0x7f16500f5410 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.893+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1650045ad0 con 0x7f16500f5410 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.893+0000 7f1645ffb700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.893+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1640041340 con 0x7f16500f5410 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.893+0000 7f1654cce700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.893+0000 7f1654cce700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 0x7f163c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:20.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:20.897+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1640012510 con 0x7f16500f5410 2024-01-18T07:21:21.103 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.102+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1650108150 con 0x7f16500f5410 2024-01-18T07:21:21.104 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.103+0000 7f1645ffb700 1 -- 172.21.15.106:0/58402789 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1640012330 con 0x7f16500f5410 2024-01-18T07:21:21.104 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:21.105 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:21.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 msgr2=0x7f163c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:21.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 0x7f163c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f164c0096a0 tx=0x7f164c006b40).stop 2024-01-18T07:21:21.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 msgr2=0x7f1650106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:21.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f1640013510 tx=0x7f164000b040).stop 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 shutdown_connections 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f163c02ff60 0x7f163c032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 --2- 172.21.15.106:0/58402789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f16500f5410 0x7f1650106720 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 >> 172.21.15.106:0/58402789 conn(0x7f16500f0b70 msgr2=0x7f165005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 shutdown_connections 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:21.105+0000 7f1657733700 1 -- 172.21.15.106:0/58402789 wait complete. 2024-01-18T07:21:21.107 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:18.912990+0000 mgr.smithi084.sihbqg (mgr.14162) 58 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:22.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:21.104506+0000 mon.smithi084 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.106:0/58402789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:22.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:22.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:20.913329+0000 mgr.smithi084.sihbqg (mgr.14162) 59 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.662+0000 7f9644439700 1 -- 172.21.15.106:0/1657782684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 msgr2=0x7f963c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:23.664 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.662+0000 7f9644439700 1 --2- 172.21.15.106:0/1657782684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c0f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f962c009fb0 tx=0x7f962c009b30).stop 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.662+0000 7f9644439700 1 -- 172.21.15.106:0/1657782684 shutdown_connections 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.662+0000 7f9644439700 1 --2- 172.21.15.106:0/1657782684 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c0f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.662+0000 7f9644439700 1 -- 172.21.15.106:0/1657782684 >> 172.21.15.106:0/1657782684 conn(0x7f963c0f1c70 msgr2=0x7f963c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 -- 172.21.15.106:0/1657782684 shutdown_connections 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 -- 172.21.15.106:0/1657782684 wait complete. 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 Processor -- start 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 -- start start 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:23.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.663+0000 7f9644439700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c069e20 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f96421d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f96421d5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:42280/0 (socket says 172.21.15.106:42280) 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f96421d5700 1 -- 172.21.15.106:0/3830959869 learned_addr learned my addr 172.21.15.106:0/3830959869 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f96421d5700 1 -- 172.21.15.106:0/3830959869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c005040 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f96421d5700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f962c014000 tx=0x7f962c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f962c0095f0 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c069fc0 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f962c007e70 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f962c014ea0 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.664+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c06a450 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.665+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f962c0079c0 con 0x7f963c0f6510 2024-01-18T07:21:23.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.665+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c059100 con 0x7f963c0f6510 2024-01-18T07:21:23.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.666+0000 7f9632ffd700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 0x7f9628032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:23.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.666+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f962c040860 con 0x7f963c0f6510 2024-01-18T07:21:23.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.666+0000 7f96419d4700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 0x7f9628032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:23.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.666+0000 7f96419d4700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 0x7f9628032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f96380096a0 tx=0x7f9638006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:23.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.670+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f962c00f950 con 0x7f963c0f6510 2024-01-18T07:21:23.877 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.876+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f963c0fbf60 con 0x7f963c0f6510 2024-01-18T07:21:23.878 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.877+0000 7f9632ffd700 1 -- 172.21.15.106:0/3830959869 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f962c010230 con 0x7f963c0f6510 2024-01-18T07:21:23.878 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:23.878 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 msgr2=0x7f9628032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 0x7f9628032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f96380096a0 tx=0x7f9638006b40).stop 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 msgr2=0x7f963c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f962c014000 tx=0x7f962c00b040).stop 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 shutdown_connections 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f962802ff60 0x7f9628032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 --2- 172.21.15.106:0/3830959869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f963c0f6510 0x7f963c06b9a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 >> 172.21.15.106:0/3830959869 conn(0x7f963c0f1c70 msgr2=0x7f963c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 shutdown_connections 2024-01-18T07:21:23.880 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:23.879+0000 7f9644439700 1 -- 172.21.15.106:0/3830959869 wait complete. 2024-01-18T07:21:23.881 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:24.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:23.878635+0000 mon.smithi084 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.106:0/3830959869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:25.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:25.275 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:22.913714+0000 mgr.smithi084.sihbqg (mgr.14162) 60 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 -- 172.21.15.106:0/3895025053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 msgr2=0x7f9e300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 --2- 172.21.15.106:0/3895025053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e300f6930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9e20009fb0 tx=0x7f9e20009b30).stop 2024-01-18T07:21:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 -- 172.21.15.106:0/3895025053 shutdown_connections 2024-01-18T07:21:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 --2- 172.21.15.106:0/3895025053 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e300f6930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:26.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 -- 172.21.15.106:0/3895025053 >> 172.21.15.106:0/3895025053 conn(0x7f9e300f1c70 msgr2=0x7f9e300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 -- 172.21.15.106:0/3895025053 shutdown_connections 2024-01-18T07:21:26.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.454+0000 7f9e35559700 1 -- 172.21.15.106:0/3895025053 wait complete. 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e35559700 1 Processor -- start 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e35559700 1 -- start start 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e35559700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e35559700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30106c20 con 0x7f9e300f6510 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e2effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e2effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37926/0 (socket says 172.21.15.106:37926) 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e2effd700 1 -- 172.21.15.106:0/230575503 learned_addr learned my addr 172.21.15.106:0/230575503 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.455+0000 7f9e2effd700 1 -- 172.21.15.106:0/230575503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e20005040 con 0x7f9e300f6510 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e2effd700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9e20013510 tx=0x7f9e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e20009320 con 0x7f9e300f6510 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e30106e20 con 0x7f9e300f6510 2024-01-18T07:21:26.458 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e20013f00 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e20014d90 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.456+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e301072b0 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e20014420 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e30045ad0 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e177fe700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 0x7f9e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e20041000 con 0x7f9e300f6510 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e2e7fc700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 0x7f9e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:26.459 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.457+0000 7f9e2e7fc700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 0x7f9e18032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9e240096a0 tx=0x7f9e24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:26.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.461+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e200095f0 con 0x7f9e300f6510 2024-01-18T07:21:26.666 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.665+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e300fbf60 con 0x7f9e300f6510 2024-01-18T07:21:26.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.666+0000 7f9e177fe700 1 -- 172.21.15.106:0/230575503 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e2001db70 con 0x7f9e300f6510 2024-01-18T07:21:26.667 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:26.667 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 msgr2=0x7f9e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 0x7f9e18032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9e240096a0 tx=0x7f9e24006b40).stop 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 msgr2=0x7f9e30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f9e20013510 tx=0x7f9e2000b040).stop 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 shutdown_connections 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9e1802ff60 0x7f9e18032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 --2- 172.21.15.106:0/230575503 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e300f6510 0x7f9e30106730 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.667+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 >> 172.21.15.106:0/230575503 conn(0x7f9e300f1c70 msgr2=0x7f9e300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.668+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 shutdown_connections 2024-01-18T07:21:26.669 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:26.668+0000 7f9e35559700 1 -- 172.21.15.106:0/230575503 wait complete. 2024-01-18T07:21:26.670 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:27.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:24.913967+0000 mgr.smithi084.sihbqg (mgr.14162) 61 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:27.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:26.667551+0000 mon.smithi084 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.106:0/230575503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:28.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:28.054 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:29.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:26.914286+0000 mgr.smithi084.sihbqg (mgr.14162) 62 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 -- 172.21.15.106:0/1793602359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 msgr2=0x7fa8080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 --2- 172.21.15.106:0/1793602359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8080f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fa7f8009fb0 tx=0x7fa7f8009b30).stop 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 -- 172.21.15.106:0/1793602359 shutdown_connections 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 --2- 172.21.15.106:0/1793602359 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8080f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 -- 172.21.15.106:0/1793602359 >> 172.21.15.106:0/1793602359 conn(0x7fa8080f0b70 msgr2=0x7fa8080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 -- 172.21.15.106:0/1793602359 shutdown_connections 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 -- 172.21.15.106:0/1793602359 wait complete. 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.248+0000 7fa80fe91700 1 Processor -- start 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80fe91700 1 -- start start 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80fe91700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80fe91700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808105ac0 con 0x7fa8080f5410 2024-01-18T07:21:29.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80dc2d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80dc2d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37946/0 (socket says 172.21.15.106:37946) 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80dc2d700 1 -- 172.21.15.106:0/2303668857 learned_addr learned my addr 172.21.15.106:0/2303668857 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80dc2d700 1 -- 172.21.15.106:0/2303668857 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8005040 con 0x7fa8080f5410 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa80dc2d700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa7f8014000 tx=0x7fa7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.249+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f80095f0 con 0x7fa8080f5410 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.250+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7f8007e70 con 0x7fa8080f5410 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.250+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa808105cc0 con 0x7fa8080f5410 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.250+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7f8014ea0 con 0x7fa8080f5410 2024-01-18T07:21:29.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.250+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808106150 con 0x7fa8080f5410 2024-01-18T07:21:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.250+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7f8025070 con 0x7fa8080f5410 2024-01-18T07:21:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.251+0000 7fa7fe7fc700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 0x7fa7f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.251+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7f8041960 con 0x7fa8080f5410 2024-01-18T07:21:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.251+0000 7fa80d42c700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 0x7fa7f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:29.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.251+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f0004fa0 con 0x7fa8080f5410 2024-01-18T07:21:29.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.252+0000 7fa80d42c700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 0x7fa7f4032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa8040096a0 tx=0x7fa804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:29.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.256+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7f8026350 con 0x7fa8080f5410 2024-01-18T07:21:29.462 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.461+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7f0005670 con 0x7fa8080f5410 2024-01-18T07:21:29.463 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.462+0000 7fa7fe7fc700 1 -- 172.21.15.106:0/2303668857 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7f8012330 con 0x7fa8080f5410 2024-01-18T07:21:29.463 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:29.463 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 msgr2=0x7fa7f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 0x7fa7f4032470 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa8040096a0 tx=0x7fa804006b40).stop 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 msgr2=0x7fa8081055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa7f8014000 tx=0x7fa7f800b040).stop 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 shutdown_connections 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7f402ffb0 0x7fa7f4032470 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 --2- 172.21.15.106:0/2303668857 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa8080f5410 0x7fa8081055d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:29.465 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 >> 172.21.15.106:0/2303668857 conn(0x7fa8080f0b70 msgr2=0x7fa8080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:29.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 shutdown_connections 2024-01-18T07:21:29.466 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:29.464+0000 7fa80fe91700 1 -- 172.21.15.106:0/2303668857 wait complete. 2024-01-18T07:21:29.466 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:29.463725+0000 mon.smithi084 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.106:0/2303668857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:30.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:30.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:28.914551+0000 mgr.smithi084.sihbqg (mgr.14162) 63 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:32.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.005+0000 7fcae884c700 1 -- 172.21.15.106:0/1530207235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 msgr2=0x7fcae40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:32.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.005+0000 7fcae884c700 1 --2- 172.21.15.106:0/1530207235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae40f66c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fcad4009fb0 tx=0x7fcad4009b30).stop 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- 172.21.15.106:0/1530207235 shutdown_connections 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 --2- 172.21.15.106:0/1530207235 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae40f66c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- 172.21.15.106:0/1530207235 >> 172.21.15.106:0/1530207235 conn(0x7fcae40f1aa0 msgr2=0x7fcae40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- 172.21.15.106:0/1530207235 shutdown_connections 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- 172.21.15.106:0/1530207235 wait complete. 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 Processor -- start 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- start start 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:32.008 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.006+0000 7fcae884c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae41069a0 con 0x7fcae40f62a0 2024-01-18T07:21:32.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:32.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37968/0 (socket says 172.21.15.106:37968) 2024-01-18T07:21:32.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae37fe700 1 -- 172.21.15.106:0/2778229645 learned_addr learned my addr 172.21.15.106:0/2778229645 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:32.010 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae37fe700 1 -- 172.21.15.106:0/2778229645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad4005040 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae37fe700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcad4013510 tx=0x7fcad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcad4009320 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae4106ba0 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcad4013f00 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcad4014dc0 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.007+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae4107030 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.008+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcad4014420 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.008+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae4045ad0 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.008+0000 7fcacbfff700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 0x7fcacc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.008+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcad4041020 con 0x7fcae40f62a0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.008+0000 7fcae2ffd700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 0x7fcacc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:32.011 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.009+0000 7fcae2ffd700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 0x7fcacc032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcad80096a0 tx=0x7fcad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:32.014 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.013+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcad401cb10 con 0x7fcae40f62a0 2024-01-18T07:21:32.222 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.221+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcae4107e80 con 0x7fcae40f62a0 2024-01-18T07:21:32.223 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.223+0000 7fcacbfff700 1 -- 172.21.15.106:0/2778229645 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcad401cb10 con 0x7fcae40f62a0 2024-01-18T07:21:32.223 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:32.223 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 msgr2=0x7fcacc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 0x7fcacc032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcad80096a0 tx=0x7fcad8006b40).stop 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 msgr2=0x7fcae41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcad4013510 tx=0x7fcad400b040).stop 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 shutdown_connections 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcacc02ff60 0x7fcacc032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 --2- 172.21.15.106:0/2778229645 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcae40f62a0 0x7fcae41064b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 >> 172.21.15.106:0/2778229645 conn(0x7fcae40f1aa0 msgr2=0x7fcae40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 shutdown_connections 2024-01-18T07:21:32.225 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:32.224+0000 7fcae884c700 1 -- 172.21.15.106:0/2778229645 wait complete. 2024-01-18T07:21:32.226 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:30.914936+0000 mgr.smithi084.sihbqg (mgr.14162) 64 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:32.223522+0000 mon.smithi084 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.106:0/2778229645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:33.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:33.637 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- 172.21.15.106:0/2548007018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 msgr2=0x7fae800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 --2- 172.21.15.106:0/2548007018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae800f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fae74009fb0 tx=0x7fae74009b30).stop 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- 172.21.15.106:0/2548007018 shutdown_connections 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 --2- 172.21.15.106:0/2548007018 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae800f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- 172.21.15.106:0/2548007018 >> 172.21.15.106:0/2548007018 conn(0x7fae800f0b70 msgr2=0x7fae800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- 172.21.15.106:0/2548007018 shutdown_connections 2024-01-18T07:21:34.774 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- 172.21.15.106:0/2548007018 wait complete. 2024-01-18T07:21:34.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 Processor -- start 2024-01-18T07:21:34.775 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.770+0000 7fae85196700 1 -- start start 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae85196700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae85196700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae80105af0 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae7ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae7ed9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:37982/0 (socket says 172.21.15.106:37982) 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae7ed9d700 1 -- 172.21.15.106:0/1077733087 learned_addr learned my addr 172.21.15.106:0/1077733087 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae7ed9d700 1 -- 172.21.15.106:0/1077733087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae74005040 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae7ed9d700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fae74013510 tx=0x7fae7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae74009320 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae80105cf0 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae74013f00 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae74014d90 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.771+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae80106180 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.772+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae740263a0 con 0x7fae800f5410 2024-01-18T07:21:34.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.772+0000 7fae677fe700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 0x7fae68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:34.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.772+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae74041020 con 0x7fae800f5410 2024-01-18T07:21:34.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.772+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae80045ad0 con 0x7fae800f5410 2024-01-18T07:21:34.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.772+0000 7fae7e59c700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 0x7fae68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:34.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.773+0000 7fae7e59c700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 0x7fae68032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae700096a0 tx=0x7fae70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:34.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.778+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae74012920 con 0x7fae800f5410 2024-01-18T07:21:34.987 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.986+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae800faf50 con 0x7fae800f5410 2024-01-18T07:21:34.988 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.987+0000 7fae677fe700 1 -- 172.21.15.106:0/1077733087 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae7402b020 con 0x7fae800f5410 2024-01-18T07:21:34.988 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:34.988 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 msgr2=0x7fae68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 0x7fae68032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae700096a0 tx=0x7fae70006b40).stop 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 msgr2=0x7fae80105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fae74013510 tx=0x7fae7400b040).stop 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 shutdown_connections 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fae6802ff60 0x7fae68032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 --2- 172.21.15.106:0/1077733087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fae800f5410 0x7fae80105600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 >> 172.21.15.106:0/1077733087 conn(0x7fae800f0b70 msgr2=0x7fae800f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:34.990 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 shutdown_connections 2024-01-18T07:21:34.991 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:34.988+0000 7fae85196700 1 -- 172.21.15.106:0/1077733087 wait complete. 2024-01-18T07:21:34.991 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:32.915242+0000 mgr.smithi084.sihbqg (mgr.14162) 65 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:36.371 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:36.371 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:34.988468+0000 mon.smithi084 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.106:0/1077733087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:35.032951+0000 mon.smithi084 (mon.0) 162 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:21:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:35.040343+0000 mon.smithi084 (mon.0) 163 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:21:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:34.915499+0000 mgr.smithi084.sihbqg (mgr.14162) 66 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 -- 172.21.15.106:0/4090953456 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 msgr2=0x7f00080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 --2- 172.21.15.106:0/4090953456 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00080f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7efff8009fb0 tx=0x7efff8009b30).stop 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 -- 172.21.15.106:0/4090953456 shutdown_connections 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 --2- 172.21.15.106:0/4090953456 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00080f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 -- 172.21.15.106:0/4090953456 >> 172.21.15.106:0/4090953456 conn(0x7f00080f09a0 msgr2=0x7f00080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 -- 172.21.15.106:0/4090953456 shutdown_connections 2024-01-18T07:21:37.517 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.515+0000 7f000759e700 1 -- 172.21.15.106:0/4090953456 wait complete. 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000759e700 1 Processor -- start 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000759e700 1 -- start start 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0008060240 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:59660/0 (socket says 172.21.15.106:59660) 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000659c700 1 -- 172.21.15.106:0/307901083 learned_addr learned my addr 172.21.15.106:0/307901083 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000659c700 1 -- 172.21.15.106:0/307901083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff8005040 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.516+0000 7f000659c700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7efff8013510 tx=0x7efff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efff8009320 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0008060440 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efff8013f00 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efff8014dc0 con 0x7f00080f51a0 2024-01-18T07:21:37.518 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0008109860 con 0x7f00080f51a0 2024-01-18T07:21:37.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.517+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0008045ad0 con 0x7f00080f51a0 2024-01-18T07:21:37.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.518+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efff80095f0 con 0x7f00080f51a0 2024-01-18T07:21:37.519 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.518+0000 7efff6ffd700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 0x7efff0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:37.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.518+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efff80410e0 con 0x7f00080f51a0 2024-01-18T07:21:37.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.518+0000 7f0005d9b700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 0x7efff0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:37.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.518+0000 7f0005d9b700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 0x7efff0032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efffc0096a0 tx=0x7efffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:37.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.522+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efff8014420 con 0x7f00080f51a0 2024-01-18T07:21:37.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.730+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f000810a070 con 0x7f00080f51a0 2024-01-18T07:21:37.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.730+0000 7efff6ffd700 1 -- 172.21.15.106:0/307901083 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efff801c920 con 0x7f00080f51a0 2024-01-18T07:21:37.731 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:37.731 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:37.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 msgr2=0x7efff0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:37.732 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 0x7efff0032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efffc0096a0 tx=0x7efffc006b40).stop 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 msgr2=0x7f00081091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7efff8013510 tx=0x7efff800b040).stop 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 shutdown_connections 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efff002ff60 0x7efff0032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 --2- 172.21.15.106:0/307901083 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f00080f51a0 0x7f00081091b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 >> 172.21.15.106:0/307901083 conn(0x7f00080f09a0 msgr2=0x7f00080f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 shutdown_connections 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:37.732+0000 7f000759e700 1 -- 172.21.15.106:0/307901083 wait complete. 2024-01-18T07:21:37.733 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:37.731022+0000 mon.smithi084 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.106:0/307901083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:39.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:39.131 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:36.915761+0000 mgr.smithi084.sihbqg (mgr.14162) 67 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:40.257 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.254+0000 7f8d5639b700 1 -- 172.21.15.106:0/1584589400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 msgr2=0x7f8d500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.254+0000 7f8d5639b700 1 --2- 172.21.15.106:0/1584589400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d500f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f8d40009fb0 tx=0x7f8d40009b30).stop 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.254+0000 7f8d5639b700 1 -- 172.21.15.106:0/1584589400 shutdown_connections 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.254+0000 7f8d5639b700 1 --2- 172.21.15.106:0/1584589400 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d500f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.254+0000 7f8d5639b700 1 -- 172.21.15.106:0/1584589400 >> 172.21.15.106:0/1584589400 conn(0x7f8d500f0b70 msgr2=0x7f8d500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 -- 172.21.15.106:0/1584589400 shutdown_connections 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 -- 172.21.15.106:0/1584589400 wait complete. 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 Processor -- start 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 -- start start 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.255+0000 7f8d5639b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d50106c20 con 0x7f8d500f5410 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:59678/0 (socket says 172.21.15.106:59678) 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4ffff700 1 -- 172.21.15.106:0/2815276322 learned_addr learned my addr 172.21.15.106:0/2815276322 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:40.258 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4ffff700 1 -- 172.21.15.106:0/2815276322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d40005040 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4ffff700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8d40013510 tx=0x7f8d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d40009320 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d50106e20 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d40013f00 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d40014d90 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.256+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d501072b0 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.257+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d40014420 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.257+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d50045ad0 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.257+0000 7f8d4cff9700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.257+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d40040fb0 con 0x7f8d500f5410 2024-01-18T07:21:40.259 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.257+0000 7f8d4f7fe700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:40.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.258+0000 7f8d4f7fe700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8d440096a0 tx=0x7f8d44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:40.262 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.262+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d400095f0 con 0x7f8d500f5410 2024-01-18T07:21:40.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.468+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d500fae60 con 0x7f8d500f5410 2024-01-18T07:21:40.469 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.469+0000 7f8d4cff9700 1 -- 172.21.15.106:0/2815276322 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8d4001db70 con 0x7f8d500f5410 2024-01-18T07:21:40.470 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:40.470 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 msgr2=0x7f8d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 0x7f8d38032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8d440096a0 tx=0x7f8d44006b40).stop 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 msgr2=0x7f8d50106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8d40013510 tx=0x7f8d4000b040).stop 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 shutdown_connections 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8d3802ff60 0x7f8d38032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 --2- 172.21.15.106:0/2815276322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8d500f5410 0x7f8d50106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 >> 172.21.15.106:0/2815276322 conn(0x7f8d500f0b70 msgr2=0x7f8d500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 shutdown_connections 2024-01-18T07:21:40.472 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:40.471+0000 7f8d5639b700 1 -- 172.21.15.106:0/2815276322 wait complete. 2024-01-18T07:21:40.473 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:38.916072+0000 mgr.smithi084.sihbqg (mgr.14162) 68 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:40.469863+0000 mon.smithi084 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.106:0/2815276322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:41.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:41.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:43.047 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 -- 172.21.15.106:0/886903657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 msgr2=0x7fc9200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 --2- 172.21.15.106:0/886903657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc9200f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc914009fb0 tx=0x7fc914009b30).stop 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 -- 172.21.15.106:0/886903657 shutdown_connections 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 --2- 172.21.15.106:0/886903657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc9200f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 -- 172.21.15.106:0/886903657 >> 172.21.15.106:0/886903657 conn(0x7fc9200f0b70 msgr2=0x7fc9200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.045+0000 7fc926f7c700 1 -- 172.21.15.106:0/886903657 shutdown_connections 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.046+0000 7fc926f7c700 1 -- 172.21.15.106:0/886903657 wait complete. 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.046+0000 7fc926f7c700 1 Processor -- start 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.046+0000 7fc926f7c700 1 -- start start 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.046+0000 7fc926f7c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.046+0000 7fc926f7c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc920105b20 con 0x7fc9200f5410 2024-01-18T07:21:43.048 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc924d18700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc924d18700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:59704/0 (socket says 172.21.15.106:59704) 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc924d18700 1 -- 172.21.15.106:0/411843759 learned_addr learned my addr 172.21.15.106:0/411843759 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc924d18700 1 -- 172.21.15.106:0/411843759 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc914005040 con 0x7fc9200f5410 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc924d18700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc914013510 tx=0x7fc91400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc914009320 con 0x7fc9200f5410 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc920105d20 con 0x7fc9200f5410 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc914013f00 con 0x7fc9200f5410 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc914014d90 con 0x7fc9200f5410 2024-01-18T07:21:43.049 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.047+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9201061b0 con 0x7fc9200f5410 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.048+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc914014420 con 0x7fc9200f5410 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.048+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc920045ad0 con 0x7fc9200f5410 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.048+0000 7fc91d7fa700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 0x7fc910032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.048+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc914041070 con 0x7fc9200f5410 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.048+0000 7fc91ffff700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 0x7fc910032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:43.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.049+0000 7fc91ffff700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 0x7fc910032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc90c0096a0 tx=0x7fc90c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:43.053 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.053+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9140095f0 con 0x7fc9200f5410 2024-01-18T07:21:43.260 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.260+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9200fae60 con 0x7fc9200f5410 2024-01-18T07:21:43.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.261+0000 7fc91d7fa700 1 -- 172.21.15.106:0/411843759 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc91401db70 con 0x7fc9200f5410 2024-01-18T07:21:43.261 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:43.261 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 msgr2=0x7fc910032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 0x7fc910032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc90c0096a0 tx=0x7fc90c006b40).stop 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 msgr2=0x7fc920105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc914013510 tx=0x7fc91400b040).stop 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 shutdown_connections 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc91002ff60 0x7fc910032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 --2- 172.21.15.106:0/411843759 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc9200f5410 0x7fc920105630 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 >> 172.21.15.106:0/411843759 conn(0x7fc9200f0b70 msgr2=0x7fc9200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 shutdown_connections 2024-01-18T07:21:43.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:43.262+0000 7fc926f7c700 1 -- 172.21.15.106:0/411843759 wait complete. 2024-01-18T07:21:43.264 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:40.916339+0000 mgr.smithi084.sihbqg (mgr.14162) 69 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:43.261491+0000 mon.smithi084 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.106:0/411843759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:44.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:44.663 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:42.916692+0000 mgr.smithi084.sihbqg (mgr.14162) 70 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.861+0000 7f54e5ebe700 1 -- 172.21.15.106:0/245390514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 msgr2=0x7f54e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.861+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/245390514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e00f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f54d4009fb0 tx=0x7f54d4009b30).stop 2024-01-18T07:21:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- 172.21.15.106:0/245390514 shutdown_connections 2024-01-18T07:21:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/245390514 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e00f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:45.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- 172.21.15.106:0/245390514 >> 172.21.15.106:0/245390514 conn(0x7f54e00f09a0 msgr2=0x7f54e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- 172.21.15.106:0/245390514 shutdown_connections 2024-01-18T07:21:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- 172.21.15.106:0/245390514 wait complete. 2024-01-18T07:21:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 Processor -- start 2024-01-18T07:21:45.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- start start 2024-01-18T07:21:45.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:45.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.862+0000 7f54e5ebe700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e0060220 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e4ebc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e4ebc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51838/0 (socket says 172.21.15.106:51838) 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e4ebc700 1 -- 172.21.15.106:0/3519660188 learned_addr learned my addr 172.21.15.106:0/3519660188 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e4ebc700 1 -- 172.21.15.106:0/3519660188 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4005040 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e4ebc700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f54d4013510 tx=0x7f54d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54d4009320 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e0060420 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54d4013f00 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54d4014dc0 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.863+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e0109880 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.864+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54d40095f0 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.864+0000 7f54dd7fa700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 0x7f54d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.864+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e0059100 con 0x7f54e00f51a0 2024-01-18T07:21:45.865 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.864+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54d4041100 con 0x7f54e00f51a0 2024-01-18T07:21:45.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.864+0000 7f54dffff700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 0x7f54d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:45.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.865+0000 7f54dffff700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 0x7f54d0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f54cc0096a0 tx=0x7f54cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:45.869 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:45.869+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54d4014420 con 0x7f54e00f51a0 2024-01-18T07:21:46.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.074+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54e00fabe0 con 0x7f54e00f51a0 2024-01-18T07:21:46.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.074+0000 7f54dd7fa700 1 -- 172.21.15.106:0/3519660188 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54d4010230 con 0x7f54e00f51a0 2024-01-18T07:21:46.075 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:46.075 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:46.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 msgr2=0x7f54d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:46.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 0x7f54d0032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f54cc0096a0 tx=0x7f54cc006b40).stop 2024-01-18T07:21:46.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 msgr2=0x7f54e01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:46.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f54d4013510 tx=0x7f54d400b040).stop 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 shutdown_connections 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f54d002ff60 0x7f54d0032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 --2- 172.21.15.106:0/3519660188 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54e00f51a0 0x7f54e01091d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 >> 172.21.15.106:0/3519660188 conn(0x7f54e00f09a0 msgr2=0x7f54e00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 shutdown_connections 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:46.076+0000 7f54e5ebe700 1 -- 172.21.15.106:0/3519660188 wait complete. 2024-01-18T07:21:46.077 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:44.917006+0000 mgr.smithi084.sihbqg (mgr.14162) 71 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:46.074999+0000 mon.smithi084 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.106:0/3519660188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:47.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:47.473 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:48.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:46.917222+0000 mgr.smithi084.sihbqg (mgr.14162) 72 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- 172.21.15.106:0/1466030457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 msgr2=0x7f5f900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 --2- 172.21.15.106:0/1466030457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5f80009fb0 tx=0x7f5f80009b30).stop 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- 172.21.15.106:0/1466030457 shutdown_connections 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 --2- 172.21.15.106:0/1466030457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- 172.21.15.106:0/1466030457 >> 172.21.15.106:0/1466030457 conn(0x7f5f900f0b70 msgr2=0x7f5f900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- 172.21.15.106:0/1466030457 shutdown_connections 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- 172.21.15.106:0/1466030457 wait complete. 2024-01-18T07:21:48.680 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 Processor -- start 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 -- start start 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.678+0000 7f5f94ace700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f94ace700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900fb820 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f8e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f8e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51858/0 (socket says 172.21.15.106:51858) 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f8e59c700 1 -- 172.21.15.106:0/2787447019 learned_addr learned my addr 172.21.15.106:0/2787447019 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f8e59c700 1 -- 172.21.15.106:0/2787447019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80005040 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f8e59c700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f5f80013510 tx=0x7f5f8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f80009320 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f900fb9c0 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f80013f00 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f80014d90 con 0x7f5f900f5410 2024-01-18T07:21:48.681 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.679+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f900fbe50 con 0x7f5f900f5410 2024-01-18T07:21:48.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.680+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f80014420 con 0x7f5f900f5410 2024-01-18T07:21:48.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.680+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90059100 con 0x7f5f900f5410 2024-01-18T07:21:48.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.680+0000 7f5f7effd700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 0x7f5f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:48.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.680+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f80041000 con 0x7f5f900f5410 2024-01-18T07:21:48.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.681+0000 7f5f8dd9b700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 0x7f5f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:48.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.681+0000 7f5f8dd9b700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 0x7f5f78032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5f840096a0 tx=0x7f5f84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:48.685 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.685+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f800095f0 con 0x7f5f900f5410 2024-01-18T07:21:48.892 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.892+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f900fae60 con 0x7f5f900f5410 2024-01-18T07:21:48.893 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.893+0000 7f5f7effd700 1 -- 172.21.15.106:0/2787447019 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f8001db70 con 0x7f5f900f5410 2024-01-18T07:21:48.893 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:48.893 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.894+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 msgr2=0x7f5f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.894+0000 7f5f94ace700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 0x7f5f78032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5f840096a0 tx=0x7f5f84006b40).stop 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.894+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 msgr2=0x7f5f900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.894+0000 7f5f94ace700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f5f80013510 tx=0x7f5f8000b040).stop 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 shutdown_connections 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5f7802ff60 0x7f5f78032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 --2- 172.21.15.106:0/2787447019 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5f900f5410 0x7f5f900fd3a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 >> 172.21.15.106:0/2787447019 conn(0x7f5f900f0b70 msgr2=0x7f5f9005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 shutdown_connections 2024-01-18T07:21:48.895 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:48.895+0000 7f5f94ace700 1 -- 172.21.15.106:0/2787447019 wait complete. 2024-01-18T07:21:48.896 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:48.893621+0000 mon.smithi084 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.106:0/2787447019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:50.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:50.287 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:48.917577+0000 mgr.smithi084.sihbqg (mgr.14162) 73 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:51.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 -- 172.21.15.106:0/730382219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 msgr2=0x7f7fc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:51.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 --2- 172.21.15.106:0/730382219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f7fac009fb0 tx=0x7f7fac009b30).stop 2024-01-18T07:21:51.438 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 -- 172.21.15.106:0/730382219 shutdown_connections 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 --2- 172.21.15.106:0/730382219 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 -- 172.21.15.106:0/730382219 >> 172.21.15.106:0/730382219 conn(0x7f7fc40f0b70 msgr2=0x7f7fc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 -- 172.21.15.106:0/730382219 shutdown_connections 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.436+0000 7f7fca077700 1 -- 172.21.15.106:0/730382219 wait complete. 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fca077700 1 Processor -- start 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fca077700 1 -- start start 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fca077700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fca077700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc40fb820 con 0x7f7fc40f5410 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fc37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51886/0 (socket says 172.21.15.106:51886) 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.437+0000 7f7fc37fe700 1 -- 172.21.15.106:0/2914060366 learned_addr learned my addr 172.21.15.106:0/2914060366 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:51.439 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fc37fe700 1 -- 172.21.15.106:0/2914060366 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac005040 con 0x7f7fc40f5410 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fc37fe700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7fac009f80 tx=0x7f7fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac009320 con 0x7f7fc40f5410 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc40fb9c0 con 0x7f7fc40f5410 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7fac013ec0 con 0x7f7fc40f5410 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac01ece0 con 0x7f7fc40f5410 2024-01-18T07:21:51.440 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.438+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc40fbe50 con 0x7f7fc40f5410 2024-01-18T07:21:51.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.439+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fac01e420 con 0x7f7fc40f5410 2024-01-18T07:21:51.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.439+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc4059100 con 0x7f7fc40f5410 2024-01-18T07:21:51.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.439+0000 7f7fbbfff700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:51.441 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.439+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fac040e20 con 0x7f7fc40f5410 2024-01-18T07:21:51.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.439+0000 7f7fc2ffd700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:51.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.440+0000 7f7fc2ffd700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 0x7f7fb0032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7fb40096a0 tx=0x7f7fb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:51.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.443+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fac01e6f0 con 0x7f7fc40f5410 2024-01-18T07:21:51.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.650+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fc40fae60 con 0x7f7fc40f5410 2024-01-18T07:21:51.651 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.651+0000 7f7fbbfff700 1 -- 172.21.15.106:0/2914060366 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fac02c2d0 con 0x7f7fc40f5410 2024-01-18T07:21:51.652 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:51.652 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:51.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 msgr2=0x7f7fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:51.653 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 0x7f7fb0032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f7fb40096a0 tx=0x7f7fb4006b40).stop 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 msgr2=0x7f7fc40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7fac009f80 tx=0x7f7fac00b040).stop 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 shutdown_connections 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7fb002ff60 0x7f7fb0032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 --2- 172.21.15.106:0/2914060366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7fc40f5410 0x7f7fc40fd3a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 >> 172.21.15.106:0/2914060366 conn(0x7f7fc40f0b70 msgr2=0x7f7fc405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 shutdown_connections 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:51.653+0000 7f7fca077700 1 -- 172.21.15.106:0/2914060366 wait complete. 2024-01-18T07:21:51.654 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:52.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:51.651881+0000 mon.smithi084 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.106:0/2914060366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:53.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:53.047 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:50.917986+0000 mgr.smithi084.sihbqg (mgr.14162) 74 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:54.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 -- 172.21.15.106:0/2453063920 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 msgr2=0x7f57940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:54.232 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 --2- 172.21.15.106:0/2453063920 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f57940f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f5788009fb0 tx=0x7f5788009b30).stop 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 -- 172.21.15.106:0/2453063920 shutdown_connections 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 --2- 172.21.15.106:0/2453063920 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f57940f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 -- 172.21.15.106:0/2453063920 >> 172.21.15.106:0/2453063920 conn(0x7f57940f1c70 msgr2=0x7f57940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 -- 172.21.15.106:0/2453063920 shutdown_connections 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.230+0000 7f5798b19700 1 -- 172.21.15.106:0/2453063920 wait complete. 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f5798b19700 1 Processor -- start 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f5798b19700 1 -- start start 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f5798b19700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f5798b19700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5794106c20 con 0x7f57940f6510 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f579259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f579259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51904/0 (socket says 172.21.15.106:51904) 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.231+0000 7f579259c700 1 -- 172.21.15.106:0/1851899540 learned_addr learned my addr 172.21.15.106:0/1851899540 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f579259c700 1 -- 172.21.15.106:0/1851899540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5788005040 con 0x7f57940f6510 2024-01-18T07:21:54.233 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f579259c700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f5788013510 tx=0x7f578800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:54.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5788009320 con 0x7f57940f6510 2024-01-18T07:21:54.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5788013f00 con 0x7f57940f6510 2024-01-18T07:21:54.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5794106e20 con 0x7f57940f6510 2024-01-18T07:21:54.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5788014d90 con 0x7f57940f6510 2024-01-18T07:21:54.234 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.232+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57941072b0 con 0x7f57940f6510 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5788014420 con 0x7f57940f6510 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794045ad0 con 0x7f57940f6510 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5782ffd700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 0x7f577c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5788041090 con 0x7f57940f6510 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5791d9b700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 0x7f577c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:54.235 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.233+0000 7f5791d9b700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 0x7f577c032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f57840096a0 tx=0x7f5784006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:54.237 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.237+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f578806d2b0 con 0x7f57940f6510 2024-01-18T07:21:54.442 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.442+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57940fbf60 con 0x7f57940f6510 2024-01-18T07:21:54.443 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.443+0000 7f5782ffd700 1 -- 172.21.15.106:0/1851899540 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57880095f0 con 0x7f57940f6510 2024-01-18T07:21:54.443 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:54.443 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 msgr2=0x7f577c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 0x7f577c032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f57840096a0 tx=0x7f5784006b40).stop 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 msgr2=0x7f5794106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f5788013510 tx=0x7f578800b040).stop 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 shutdown_connections 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f577c02ffb0 0x7f577c032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 --2- 172.21.15.106:0/1851899540 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f57940f6510 0x7f5794106730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 >> 172.21.15.106:0/1851899540 conn(0x7f57940f1c70 msgr2=0x7f57940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.444+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 shutdown_connections 2024-01-18T07:21:54.445 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:54.445+0000 7f5798b19700 1 -- 172.21.15.106:0/1851899540 wait complete. 2024-01-18T07:21:54.446 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:52.918242+0000 mgr.smithi084.sihbqg (mgr.14162) 75 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:54.443469+0000 mon.smithi084 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.106:0/1851899540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:55.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:55.839 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:57.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 -- 172.21.15.106:0/4054651211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 msgr2=0x7fedf40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:57.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/4054651211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf40f90f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7feddc009fb0 tx=0x7feddc009b30).stop 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 -- 172.21.15.106:0/4054651211 shutdown_connections 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/4054651211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf40f90f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 -- 172.21.15.106:0/4054651211 >> 172.21.15.106:0/4054651211 conn(0x7fedf40f0b70 msgr2=0x7fedf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 -- 172.21.15.106:0/4054651211 shutdown_connections 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 -- 172.21.15.106:0/4054651211 wait complete. 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.027+0000 7fedfa2a9700 1 Processor -- start 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedfa2a9700 1 -- start start 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedfa2a9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedfa2a9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4106bd0 con 0x7fedf4060280 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedf3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedf3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:36194/0 (socket says 172.21.15.106:36194) 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedf3fff700 1 -- 172.21.15.106:0/3503177362 learned_addr learned my addr 172.21.15.106:0/3503177362 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedf3fff700 1 -- 172.21.15.106:0/3503177362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc005040 con 0x7fedf4060280 2024-01-18T07:21:57.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.028+0000 7fedf3fff700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feddc013510 tx=0x7feddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feddc009320 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf4106dd0 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feddc013f00 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feddc014dc0 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf4107260 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.029+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feddc014420 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.030+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf4045ad0 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.030+0000 7fedf0ff9700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 0x7fede0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.030+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feddc041110 con 0x7fedf4060280 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.030+0000 7fedf37fe700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 0x7fede0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:57.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.030+0000 7fedf37fe700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 0x7fede0032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fede40096a0 tx=0x7fede4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:57.037 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.037+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feddc0095f0 con 0x7fedf4060280 2024-01-18T07:21:57.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.244+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedf4108110 con 0x7fedf4060280 2024-01-18T07:21:57.246 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.245+0000 7fedf0ff9700 1 -- 172.21.15.106:0/3503177362 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feddc01cb10 con 0x7fedf4060280 2024-01-18T07:21:57.246 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:21:57.246 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 msgr2=0x7fede0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 0x7fede0032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fede40096a0 tx=0x7fede4006b40).stop 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 msgr2=0x7fedf41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7feddc013510 tx=0x7feddc00b040).stop 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 shutdown_connections 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fede002ff60 0x7fede0032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 --2- 172.21.15.106:0/3503177362 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fedf4060280 0x7fedf41066e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 >> 172.21.15.106:0/3503177362 conn(0x7fedf40f0b70 msgr2=0x7fedf40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:57.248 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 shutdown_connections 2024-01-18T07:21:57.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:57.247+0000 7fedfa2a9700 1 -- 172.21.15.106:0/3503177362 wait complete. 2024-01-18T07:21:57.249 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:21:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:54.918533+0000 mgr.smithi084.sihbqg (mgr.14162) 76 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:21:57.246854+0000 mon.smithi084 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.106:0/3503177362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:21:58.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:21:58.689 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:21:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:21:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:56.918818+0000 mgr.smithi084.sihbqg (mgr.14162) 77 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 -- 172.21.15.106:0/2851305511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 msgr2=0x7fc5a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 --2- 172.21.15.106:0/2851305511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a40f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fc594009fb0 tx=0x7fc594009b30).stop 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 -- 172.21.15.106:0/2851305511 shutdown_connections 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 --2- 172.21.15.106:0/2851305511 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a40f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 -- 172.21.15.106:0/2851305511 >> 172.21.15.106:0/2851305511 conn(0x7fc5a40f0b70 msgr2=0x7fc5a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 -- 172.21.15.106:0/2851305511 shutdown_connections 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.857+0000 7fc5abe49700 1 -- 172.21.15.106:0/2851305511 wait complete. 2024-01-18T07:21:59.862 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.858+0000 7fc5abe49700 1 Processor -- start 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.858+0000 7fc5abe49700 1 -- start start 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.858+0000 7fc5abe49700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.858+0000 7fc5abe49700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a4107240 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5a9be5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5a9be5700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:36222/0 (socket says 172.21.15.106:36222) 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5a9be5700 1 -- 172.21.15.106:0/1727474972 learned_addr learned my addr 172.21.15.106:0/1727474972 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5a9be5700 1 -- 172.21.15.106:0/1727474972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc594005040 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5a9be5700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc594013510 tx=0x7fc59400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc594009320 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a4060220 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc594013f00 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc594014d90 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.859+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a410bd40 con 0x7fc5a40f5410 2024-01-18T07:21:59.863 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.860+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc5940095f0 con 0x7fc5a40f5410 2024-01-18T07:21:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.860+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a4045ad0 con 0x7fc5a40f5410 2024-01-18T07:21:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.860+0000 7fc59a7fc700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:21:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.860+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc5940410c0 con 0x7fc5a40f5410 2024-01-18T07:21:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.860+0000 7fc5a93e4700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:21:59.864 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.861+0000 7fc5a93e4700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 0x7fc590032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc5a00096a0 tx=0x7fc5a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:21:59.866 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:21:59.865+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc594014420 con 0x7fc5a40f5410 2024-01-18T07:22:00.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.073+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5a40fae60 con 0x7fc5a40f5410 2024-01-18T07:22:00.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.073+0000 7fc59a7fc700 1 -- 172.21.15.106:0/1727474972 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc594010230 con 0x7fc5a40f5410 2024-01-18T07:22:00.075 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:00.075 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:00.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 msgr2=0x7fc590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:00.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 0x7fc590032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc5a00096a0 tx=0x7fc5a0006b40).stop 2024-01-18T07:22:00.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 msgr2=0x7fc5a410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:00.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc594013510 tx=0x7fc59400b040).stop 2024-01-18T07:22:00.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 shutdown_connections 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc59002ff60 0x7fc590032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 --2- 172.21.15.106:0/1727474972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5a40f5410 0x7fc5a410b850 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 >> 172.21.15.106:0/1727474972 conn(0x7fc5a40f0b70 msgr2=0x7fc5a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 shutdown_connections 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:00.075+0000 7fc5abe49700 1 -- 172.21.15.106:0/1727474972 wait complete. 2024-01-18T07:22:00.077 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:21:58.919191+0000 mgr.smithi084.sihbqg (mgr.14162) 78 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:00.075011+0000 mon.smithi084 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.106:0/1727474972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:01.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:01.474 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:02.586 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- 172.21.15.106:0/1895566689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 msgr2=0x7f21e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/1895566689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e40f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f21d4009fb0 tx=0x7f21d4009b30).stop 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- 172.21.15.106:0/1895566689 shutdown_connections 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/1895566689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e40f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- 172.21.15.106:0/1895566689 >> 172.21.15.106:0/1895566689 conn(0x7f21e40f0b70 msgr2=0x7f21e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- 172.21.15.106:0/1895566689 shutdown_connections 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- 172.21.15.106:0/1895566689 wait complete. 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 Processor -- start 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.585+0000 7f21ebb4b700 1 -- start start 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21ebb4b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:02.587 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21ebb4b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4105b20 con 0x7f21e40f5410 2024-01-18T07:22:02.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21e98e7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:02.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21e98e7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:36242/0 (socket says 172.21.15.106:36242) 2024-01-18T07:22:02.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21e98e7700 1 -- 172.21.15.106:0/2685182230 learned_addr learned my addr 172.21.15.106:0/2685182230 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:02.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21e98e7700 1 -- 172.21.15.106:0/2685182230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d4005040 con 0x7f21e40f5410 2024-01-18T07:22:02.589 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21e98e7700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f21d4013510 tx=0x7f21d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.586+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d4009320 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e4105d20 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21d4013f00 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21d4014d90 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e41061b0 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21d40095f0 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.587+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21e4045ad0 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.588+0000 7f21da7fc700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 0x7f21d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.588+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21d4041000 con 0x7f21e40f5410 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.588+0000 7f21e90e6700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 0x7f21d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:02.590 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.588+0000 7f21e90e6700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 0x7f21d0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f21e00096a0 tx=0x7f21e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:02.593 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.593+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21d4014420 con 0x7f21e40f5410 2024-01-18T07:22:02.805 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.804+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21e40fae60 con 0x7f21e40f5410 2024-01-18T07:22:02.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.805+0000 7f21da7fc700 1 -- 172.21.15.106:0/2685182230 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21d401d7f0 con 0x7f21e40f5410 2024-01-18T07:22:02.806 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:02.806 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:02.807 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 msgr2=0x7f21d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 0x7f21d0032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f21e00096a0 tx=0x7f21e0006b40).stop 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 msgr2=0x7f21e4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f21d4013510 tx=0x7f21d400b040).stop 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 shutdown_connections 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f21d002ff60 0x7f21d0032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 --2- 172.21.15.106:0/2685182230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21e40f5410 0x7f21e4105630 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 >> 172.21.15.106:0/2685182230 conn(0x7f21e40f0b70 msgr2=0x7f21e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 shutdown_connections 2024-01-18T07:22:02.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:02.806+0000 7f21ebb4b700 1 -- 172.21.15.106:0/2685182230 wait complete. 2024-01-18T07:22:02.809 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:00.919561+0000 mgr.smithi084.sihbqg (mgr.14162) 79 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:02.806164+0000 mon.smithi084 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.106:0/2685182230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:04.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:04.199 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 -- 172.21.15.106:0/91564158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 msgr2=0x7f6a9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 --2- 172.21.15.106:0/91564158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f6a8c009fb0 tx=0x7f6a8c009b30).stop 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 -- 172.21.15.106:0/91564158 shutdown_connections 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 --2- 172.21.15.106:0/91564158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 -- 172.21.15.106:0/91564158 >> 172.21.15.106:0/91564158 conn(0x7f6a9c0f0b70 msgr2=0x7f6a9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 -- 172.21.15.106:0/91564158 shutdown_connections 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.361+0000 7f6aa4024700 1 -- 172.21.15.106:0/91564158 wait complete. 2024-01-18T07:22:05.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa4024700 1 Processor -- start 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa4024700 1 -- start start 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa4024700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa4024700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c106c10 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa1dc0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa1dc0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35828/0 (socket says 172.21.15.106:35828) 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa1dc0700 1 -- 172.21.15.106:0/1231528178 learned_addr learned my addr 172.21.15.106:0/1231528178 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa1dc0700 1 -- 172.21.15.106:0/1231528178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c005040 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.362+0000 7f6aa1dc0700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6a8c014000 tx=0x7f6a8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a8c0095f0 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c106e10 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a8c007e70 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a8c014ea0 con 0x7f6a9c0f5410 2024-01-18T07:22:05.365 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c1072a0 con 0x7f6a9c0f5410 2024-01-18T07:22:05.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.363+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a8c0079c0 con 0x7f6a9c0f5410 2024-01-18T07:22:05.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.364+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9c045ad0 con 0x7f6a9c0f5410 2024-01-18T07:22:05.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.364+0000 7f6a927fc700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 0x7f6a88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:05.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.364+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a8c040860 con 0x7f6a9c0f5410 2024-01-18T07:22:05.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.364+0000 7f6aa15bf700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 0x7f6a88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:05.367 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.364+0000 7f6aa15bf700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 0x7f6a88032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6a980096a0 tx=0x7f6a98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:05.369 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.369+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a8c00f950 con 0x7f6a9c0f5410 2024-01-18T07:22:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:02.919957+0000 mgr.smithi084.sihbqg (mgr.14162) 80 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:05.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.575+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a9c108150 con 0x7f6a9c0f5410 2024-01-18T07:22:05.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.576+0000 7f6a927fc700 1 -- 172.21.15.106:0/1231528178 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a8c010230 con 0x7f6a9c0f5410 2024-01-18T07:22:05.577 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:05.577 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:05.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 msgr2=0x7f6a88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:05.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 0x7f6a88032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6a980096a0 tx=0x7f6a98006b40).stop 2024-01-18T07:22:05.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 msgr2=0x7f6a9c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:05.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6a8c014000 tx=0x7f6a8c00b040).stop 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 shutdown_connections 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6a8802ff60 0x7f6a88032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 --2- 172.21.15.106:0/1231528178 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6a9c0f5410 0x7f6a9c106720 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 >> 172.21.15.106:0/1231528178 conn(0x7f6a9c0f0b70 msgr2=0x7f6a9c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 shutdown_connections 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:05.577+0000 7f6aa4024700 1 -- 172.21.15.106:0/1231528178 wait complete. 2024-01-18T07:22:05.579 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:06.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:05.577221+0000 mon.smithi084 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.106:0/1231528178' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:06.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:06.976 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:04.920321+0000 mgr.smithi084.sihbqg (mgr.14162) 81 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 -- 172.21.15.106:0/3611085120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 msgr2=0x7fa7700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 --2- 172.21.15.106:0/3611085120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa7700f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fa764009fb0 tx=0x7fa764009b30).stop 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 -- 172.21.15.106:0/3611085120 shutdown_connections 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 --2- 172.21.15.106:0/3611085120 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa7700f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 -- 172.21.15.106:0/3611085120 >> 172.21.15.106:0/3611085120 conn(0x7fa7700f1c70 msgr2=0x7fa7700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 -- 172.21.15.106:0/3611085120 shutdown_connections 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.138+0000 7fa775e7f700 1 -- 172.21.15.106:0/3611085120 wait complete. 2024-01-18T07:22:08.140 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.139+0000 7fa775e7f700 1 Processor -- start 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.139+0000 7fa775e7f700 1 -- start start 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.139+0000 7fa775e7f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.139+0000 7fa775e7f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa770106c20 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa76f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa76f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35846/0 (socket says 172.21.15.106:35846) 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa76f7fe700 1 -- 172.21.15.106:0/653853492 learned_addr learned my addr 172.21.15.106:0/653853492 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa76f7fe700 1 -- 172.21.15.106:0/653853492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa764005040 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa76f7fe700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa764013510 tx=0x7fa76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa764009320 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa770106e20 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa764013f00 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa764014d90 con 0x7fa7700f6510 2024-01-18T07:22:08.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.140+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7701072b0 con 0x7fa7700f6510 2024-01-18T07:22:08.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.141+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7640095f0 con 0x7fa7700f6510 2024-01-18T07:22:08.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.141+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa770045ad0 con 0x7fa7700f6510 2024-01-18T07:22:08.143 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.141+0000 7fa757fff700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 0x7fa758032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:08.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.141+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7640410a0 con 0x7fa7700f6510 2024-01-18T07:22:08.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.141+0000 7fa76effd700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 0x7fa758032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:08.144 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.142+0000 7fa76effd700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 0x7fa758032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa7600096a0 tx=0x7fa760006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:08.147 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.146+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa76401d7f0 con 0x7fa7700f6510 2024-01-18T07:22:08.356 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.355+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7700fbf60 con 0x7fa7700f6510 2024-01-18T07:22:08.357 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.355+0000 7fa757fff700 1 -- 172.21.15.106:0/653853492 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa764010230 con 0x7fa7700f6510 2024-01-18T07:22:08.357 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:08.357 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:08.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 msgr2=0x7fa758032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:08.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 0x7fa758032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa7600096a0 tx=0x7fa760006b40).stop 2024-01-18T07:22:08.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 msgr2=0x7fa770106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:08.358 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa764013510 tx=0x7fa76400b040).stop 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 shutdown_connections 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa75802ff60 0x7fa758032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 --2- 172.21.15.106:0/653853492 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7700f6510 0x7fa770106730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 >> 172.21.15.106:0/653853492 conn(0x7fa7700f1c70 msgr2=0x7fa7700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 shutdown_connections 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:08.357+0000 7fa775e7f700 1 -- 172.21.15.106:0/653853492 wait complete. 2024-01-18T07:22:08.359 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:06.920589+0000 mgr.smithi084.sihbqg (mgr.14162) 82 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:08.357223+0000 mon.smithi084 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.106:0/653853492' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:09.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:09.756 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.934+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/1237750781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 msgr2=0x7f4a180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.934+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/1237750781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a180f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f4a0c009fb0 tx=0x7f4a0c009b30).stop 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.934+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/1237750781 shutdown_connections 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.934+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/1237750781 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a180f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.934+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/1237750781 >> 172.21.15.106:0/1237750781 conn(0x7f4a180f0b70 msgr2=0x7f4a180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/1237750781 shutdown_connections 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/1237750781 wait complete. 2024-01-18T07:22:10.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 Processor -- start 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 -- start start 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.935+0000 7f4a1ccb9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a18105b20 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35878/0 (socket says 172.21.15.106:35878) 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1659c700 1 -- 172.21.15.106:0/827686919 learned_addr learned my addr 172.21.15.106:0/827686919 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1659c700 1 -- 172.21.15.106:0/827686919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c005040 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1659c700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f4a0c013510 tx=0x7f4a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c009320 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a0c013f00 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a18105d20 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c014d90 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.936+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a181061b0 con 0x7f4a180f5410 2024-01-18T07:22:10.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.937+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a0c0095f0 con 0x7f4a180f5410 2024-01-18T07:22:10.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.937+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a18045ad0 con 0x7f4a180f5410 2024-01-18T07:22:10.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.937+0000 7f4a06ffd700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 0x7f4a00032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:10.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.937+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a0c0410a0 con 0x7f4a180f5410 2024-01-18T07:22:10.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.938+0000 7f4a15d9b700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 0x7f4a00032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:10.941 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.938+0000 7f4a15d9b700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 0x7f4a00032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4a080096a0 tx=0x7f4a08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:10.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:10.942+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a0c01d380 con 0x7f4a180f5410 2024-01-18T07:22:11.149 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.148+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a180faf50 con 0x7f4a180f5410 2024-01-18T07:22:11.150 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.149+0000 7f4a06ffd700 1 -- 172.21.15.106:0/827686919 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a0c010230 con 0x7f4a180f5410 2024-01-18T07:22:11.150 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:11.150 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 msgr2=0x7f4a00032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 0x7f4a00032470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4a080096a0 tx=0x7f4a08006b40).stop 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 msgr2=0x7f4a18105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f4a0c013510 tx=0x7f4a0c00b040).stop 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 shutdown_connections 2024-01-18T07:22:11.152 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.150+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4a0002ffb0 0x7f4a00032470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.151+0000 7f4a1ccb9700 1 --2- 172.21.15.106:0/827686919 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a180f5410 0x7f4a18105630 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.151+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 >> 172.21.15.106:0/827686919 conn(0x7f4a180f0b70 msgr2=0x7f4a180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.151+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 shutdown_connections 2024-01-18T07:22:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:11.151+0000 7f4a1ccb9700 1 -- 172.21.15.106:0/827686919 wait complete. 2024-01-18T07:22:11.153 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:08.920930+0000 mgr.smithi084.sihbqg (mgr.14162) 83 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:11.150485+0000 mon.smithi084 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.106:0/827686919' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:12.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:12.562 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:10.921261+0000 mgr.smithi084.sihbqg (mgr.14162) 84 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:13.715 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 -- 172.21.15.106:0/943352324 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 msgr2=0x7fc2740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:13.715 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 --2- 172.21.15.106:0/943352324 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc2740f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc25c009fb0 tx=0x7fc25c009b30).stop 2024-01-18T07:22:13.715 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 -- 172.21.15.106:0/943352324 shutdown_connections 2024-01-18T07:22:13.715 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 --2- 172.21.15.106:0/943352324 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc2740f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 -- 172.21.15.106:0/943352324 >> 172.21.15.106:0/943352324 conn(0x7fc2740f1c70 msgr2=0x7fc2740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 -- 172.21.15.106:0/943352324 shutdown_connections 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.713+0000 7fc27a52a700 1 -- 172.21.15.106:0/943352324 wait complete. 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc27a52a700 1 Processor -- start 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc27a52a700 1 -- start start 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc27a52a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc27a52a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274106c20 con 0x7fc2740f6510 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc273fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc273fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35894/0 (socket says 172.21.15.106:35894) 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc273fff700 1 -- 172.21.15.106:0/350899164 learned_addr learned my addr 172.21.15.106:0/350899164 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:13.716 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.714+0000 7fc273fff700 1 -- 172.21.15.106:0/350899164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c005040 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc273fff700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc25c013510 tx=0x7fc25c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc25c009320 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274106e20 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc25c013f00 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc25c014d90 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2741072b0 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.715+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc25c0095f0 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.716+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc274045ad0 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.716+0000 7fc270ff9700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.716+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc25c040fc0 con 0x7fc2740f6510 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.716+0000 7fc2737fe700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:13.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.716+0000 7fc2737fe700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 0x7fc260032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:13.721 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.720+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc25c014420 con 0x7fc2740f6510 2024-01-18T07:22:13.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.928+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2740fbf60 con 0x7fc2740f6510 2024-01-18T07:22:13.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.929+0000 7fc270ff9700 1 -- 172.21.15.106:0/350899164 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc25c010230 con 0x7fc2740f6510 2024-01-18T07:22:13.930 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:13.930 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:13.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 msgr2=0x7fc260032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:13.932 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 0x7fc260032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc2640096a0 tx=0x7fc264006b40).stop 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 msgr2=0x7fc274106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fc25c013510 tx=0x7fc25c00b040).stop 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 shutdown_connections 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc26002ff60 0x7fc260032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 --2- 172.21.15.106:0/350899164 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc2740f6510 0x7fc274106730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 >> 172.21.15.106:0/350899164 conn(0x7fc2740f1c70 msgr2=0x7fc2740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 shutdown_connections 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:13.931+0000 7fc27a52a700 1 -- 172.21.15.106:0/350899164 wait complete. 2024-01-18T07:22:13.933 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:14.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:12.921595+0000 mgr.smithi084.sihbqg (mgr.14162) 85 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:15.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:15.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:13.930711+0000 mon.smithi084 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.106:0/350899164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:16.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- 172.21.15.106:0/1228688435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 msgr2=0x7fbd7c0f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:16.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 --2- 172.21.15.106:0/1228688435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f91b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbd70009fb0 tx=0x7fbd70009b30).stop 2024-01-18T07:22:16.507 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- 172.21.15.106:0/1228688435 shutdown_connections 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 --2- 172.21.15.106:0/1228688435 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f91b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- 172.21.15.106:0/1228688435 >> 172.21.15.106:0/1228688435 conn(0x7fbd7c0f0b70 msgr2=0x7fbd7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- 172.21.15.106:0/1228688435 shutdown_connections 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- 172.21.15.106:0/1228688435 wait complete. 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 Processor -- start 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.505+0000 7fbd81173700 1 -- start start 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd81173700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd81173700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c0f50b0 con 0x7fbd7c0601c0 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd7ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd7ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:44778/0 (socket says 172.21.15.106:44778) 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd7ad9d700 1 -- 172.21.15.106:0/583802466 learned_addr learned my addr 172.21.15.106:0/583802466 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd7ad9d700 1 -- 172.21.15.106:0/583802466 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70005040 con 0x7fbd7c0601c0 2024-01-18T07:22:16.508 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd7ad9d700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbd70013510 tx=0x7fbd7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:16.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd70009320 con 0x7fbd7c0601c0 2024-01-18T07:22:16.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7c0f5250 con 0x7fbd7c0601c0 2024-01-18T07:22:16.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd70013f00 con 0x7fbd7c0601c0 2024-01-18T07:22:16.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd70014dc0 con 0x7fbd7c0601c0 2024-01-18T07:22:16.509 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.506+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c0f56e0 con 0x7fbd7c0601c0 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.507+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd700095f0 con 0x7fbd7c0601c0 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.507+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7c059100 con 0x7fbd7c0601c0 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.507+0000 7fbd637fe700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 0x7fbd64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.507+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd700410c0 con 0x7fbd7c0601c0 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.507+0000 7fbd7a59c700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 0x7fbd64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:16.510 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.508+0000 7fbd7a59c700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 0x7fbd64032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbd6c0096a0 tx=0x7fbd6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:16.513 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.513+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd70014420 con 0x7fbd7c0601c0 2024-01-18T07:22:16.717 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.716+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd7c0fbf50 con 0x7fbd7c0601c0 2024-01-18T07:22:16.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd637fe700 1 -- 172.21.15.106:0/583802466 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd70010230 con 0x7fbd7c0601c0 2024-01-18T07:22:16.718 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:16.718 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 msgr2=0x7fbd64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 0x7fbd64032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fbd6c0096a0 tx=0x7fbd6c006b40).stop 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 msgr2=0x7fbd7c0f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbd70013510 tx=0x7fbd7000b040).stop 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 shutdown_connections 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fbd6402ff60 0x7fbd64032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 --2- 172.21.15.106:0/583802466 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fbd7c0601c0 0x7fbd7c0f6c30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 >> 172.21.15.106:0/583802466 conn(0x7fbd7c0f0b70 msgr2=0x7fbd7c0f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.718+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 shutdown_connections 2024-01-18T07:22:16.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:16.719+0000 7fbd81173700 1 -- 172.21.15.106:0/583802466 wait complete. 2024-01-18T07:22:16.721 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:14.921851+0000 mgr.smithi084.sihbqg (mgr.14162) 86 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:16.718427+0000 mon.smithi084 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.106:0/583802466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:18.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:18.105 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:19.275 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 -- 172.21.15.106:0/3272320291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 msgr2=0x7fca2c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:19.275 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 --2- 172.21.15.106:0/3272320291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c0f66c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fca1c009fb0 tx=0x7fca1c009b30).stop 2024-01-18T07:22:19.275 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 -- 172.21.15.106:0/3272320291 shutdown_connections 2024-01-18T07:22:19.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 --2- 172.21.15.106:0/3272320291 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c0f66c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:19.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 -- 172.21.15.106:0/3272320291 >> 172.21.15.106:0/3272320291 conn(0x7fca2c0f1aa0 msgr2=0x7fca2c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:19.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 -- 172.21.15.106:0/3272320291 shutdown_connections 2024-01-18T07:22:19.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.275+0000 7fca32bac700 1 -- 172.21.15.106:0/3272320291 wait complete. 2024-01-18T07:22:19.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca32bac700 1 Processor -- start 2024-01-18T07:22:19.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca32bac700 1 -- start start 2024-01-18T07:22:19.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca32bac700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:19.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca32bac700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2c1069a0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.277 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca31baa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca31baa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:44800/0 (socket says 172.21.15.106:44800) 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca31baa700 1 -- 172.21.15.106:0/1062448950 learned_addr learned my addr 172.21.15.106:0/1062448950 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.276+0000 7fca31baa700 1 -- 172.21.15.106:0/1062448950 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1c005040 con 0x7fca2c0f62a0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca31baa700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fca1c013510 tx=0x7fca1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c009320 con 0x7fca2c0f62a0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca1c013f00 con 0x7fca2c0f62a0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c106ba0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca1c014dc0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.278 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.277+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2c107030 con 0x7fca2c0f62a0 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca1c0095f0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2c045ad0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca227fc700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 0x7fca18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca1c041080 con 0x7fca2c0f62a0 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca313a9700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 0x7fca18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:19.279 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.278+0000 7fca313a9700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 0x7fca18032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:19.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.282+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca1c01c790 con 0x7fca2c0f62a0 2024-01-18T07:22:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:16.922187+0000 mgr.smithi084.sihbqg (mgr.14162) 87 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:19.486 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.486+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca2c107ee0 con 0x7fca2c0f62a0 2024-01-18T07:22:19.487 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.487+0000 7fca227fc700 1 -- 172.21.15.106:0/1062448950 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca1c010230 con 0x7fca2c0f62a0 2024-01-18T07:22:19.487 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:19.488 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 msgr2=0x7fca18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 0x7fca18032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fca280096a0 tx=0x7fca28006b40).stop 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 msgr2=0x7fca2c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fca1c013510 tx=0x7fca1c00b040).stop 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 shutdown_connections 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca1802ff60 0x7fca18032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 --2- 172.21.15.106:0/1062448950 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca2c0f62a0 0x7fca2c1064b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 >> 172.21.15.106:0/1062448950 conn(0x7fca2c0f1aa0 msgr2=0x7fca2c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 shutdown_connections 2024-01-18T07:22:19.489 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:19.489+0000 7fca32bac700 1 -- 172.21.15.106:0/1062448950 wait complete. 2024-01-18T07:22:19.490 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:19.487564+0000 mon.smithi084 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.106:0/1062448950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:20.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:20.874 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:18.922496+0000 mgr.smithi084.sihbqg (mgr.14162) 88 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:21.992 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- 172.21.15.106:0/3500287223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 msgr2=0x7ff9d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 --2- 172.21.15.106:0/3500287223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d40f55c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff9d0009fb0 tx=0x7ff9d0009b30).stop 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- 172.21.15.106:0/3500287223 shutdown_connections 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 --2- 172.21.15.106:0/3500287223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d40f55c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- 172.21.15.106:0/3500287223 >> 172.21.15.106:0/3500287223 conn(0x7ff9d40f09a0 msgr2=0x7ff9d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- 172.21.15.106:0/3500287223 shutdown_connections 2024-01-18T07:22:21.993 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- 172.21.15.106:0/3500287223 wait complete. 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 Processor -- start 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- start start 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9db064700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4108bc0 con 0x7ff9d40f51a0 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9da062700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.989+0000 7ff9da062700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:44830/0 (socket says 172.21.15.106:44830) 2024-01-18T07:22:21.994 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9da062700 1 -- 172.21.15.106:0/3480718523 learned_addr learned my addr 172.21.15.106:0/3480718523 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9da062700 1 -- 172.21.15.106:0/3480718523 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0005040 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9da062700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ff9d0013510 tx=0x7ff9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d0009320 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9d0013f00 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d4108dc0 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9d0014dc0 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.990+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d4109250 con 0x7ff9d40f51a0 2024-01-18T07:22:21.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.991+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9d00095f0 con 0x7ff9d40f51a0 2024-01-18T07:22:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.991+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d4045ad0 con 0x7ff9d40f51a0 2024-01-18T07:22:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.991+0000 7ff9caffd700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 0x7ff9c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.991+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9d0041000 con 0x7ff9d40f51a0 2024-01-18T07:22:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.991+0000 7ff9d9861700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 0x7ff9c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:21.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.992+0000 7ff9d9861700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 0x7ff9c0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff9c40096a0 tx=0x7ff9c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:21.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:21.996+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9d0014420 con 0x7ff9d40f51a0 2024-01-18T07:22:22.202 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.202+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9d410a100 con 0x7ff9d40f51a0 2024-01-18T07:22:22.203 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.203+0000 7ff9caffd700 1 -- 172.21.15.106:0/3480718523 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9d0010230 con 0x7ff9d40f51a0 2024-01-18T07:22:22.203 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:22.203 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:22.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 msgr2=0x7ff9c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:22.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 0x7ff9c0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff9c40096a0 tx=0x7ff9c4006b40).stop 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 msgr2=0x7ff9d41086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7ff9d0013510 tx=0x7ff9d000b040).stop 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 shutdown_connections 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9c002ff60 0x7ff9c0032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 --2- 172.21.15.106:0/3480718523 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9d40f51a0 0x7ff9d41086d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 >> 172.21.15.106:0/3480718523 conn(0x7ff9d40f09a0 msgr2=0x7ff9d405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 shutdown_connections 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:22.204+0000 7ff9db064700 1 -- 172.21.15.106:0/3480718523 wait complete. 2024-01-18T07:22:22.205 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:20.922855+0000 mgr.smithi084.sihbqg (mgr.14162) 89 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:22.203208+0000 mon.smithi084 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.106:0/3480718523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:23.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:23.601 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 -- 172.21.15.106:0/1443816475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 msgr2=0x7fc0f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 --2- 172.21.15.106:0/1443816475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f80f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc0e8009fb0 tx=0x7fc0e8009b30).stop 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 -- 172.21.15.106:0/1443816475 shutdown_connections 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 --2- 172.21.15.106:0/1443816475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f80f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 -- 172.21.15.106:0/1443816475 >> 172.21.15.106:0/1443816475 conn(0x7fc0f80f0b70 msgr2=0x7fc0f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 -- 172.21.15.106:0/1443816475 shutdown_connections 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.754+0000 7fc0fc948700 1 -- 172.21.15.106:0/1443816475 wait complete. 2024-01-18T07:22:24.755 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0fc948700 1 Processor -- start 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0fc948700 1 -- start start 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0fc948700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0fc948700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f8107210 con 0x7fc0f80f5410 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0f659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0f659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:44862/0 (socket says 172.21.15.106:44862) 2024-01-18T07:22:24.756 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0f659c700 1 -- 172.21.15.106:0/252596859 learned_addr learned my addr 172.21.15.106:0/252596859 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:24.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0f659c700 1 -- 172.21.15.106:0/252596859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e8005040 con 0x7fc0f80f5410 2024-01-18T07:22:24.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0f659c700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc0e8013510 tx=0x7fc0e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:24.757 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.755+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e8009320 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f8060220 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0e8013f00 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0e8014d90 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f810bd10 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0e80095f0 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f8045ad0 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.756+0000 7fc0e6ffd700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 0x7fc0e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.757+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0e80410c0 con 0x7fc0f80f5410 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.757+0000 7fc0f5d9b700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 0x7fc0e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:24.758 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.757+0000 7fc0f5d9b700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 0x7fc0e0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc0ec0096a0 tx=0x7fc0ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:24.761 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.761+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0e8014420 con 0x7fc0f80f5410 2024-01-18T07:22:24.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.966+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0f80fae60 con 0x7fc0f80f5410 2024-01-18T07:22:24.967 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.967+0000 7fc0e6ffd700 1 -- 172.21.15.106:0/252596859 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0e8010230 con 0x7fc0f80f5410 2024-01-18T07:22:24.967 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:24.968 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:24.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 msgr2=0x7fc0e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:24.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 0x7fc0e0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc0ec0096a0 tx=0x7fc0ec006b40).stop 2024-01-18T07:22:24.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 msgr2=0x7fc0f810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:24.969 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fc0e8013510 tx=0x7fc0e800b040).stop 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 shutdown_connections 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc0e002ff60 0x7fc0e0032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 --2- 172.21.15.106:0/252596859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc0f80f5410 0x7fc0f810b820 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 >> 172.21.15.106:0/252596859 conn(0x7fc0f80f0b70 msgr2=0x7fc0f80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 shutdown_connections 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:24.969+0000 7fc0fc948700 1 -- 172.21.15.106:0/252596859 wait complete. 2024-01-18T07:22:24.970 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:22.923193+0000 mgr.smithi084.sihbqg (mgr.14162) 90 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:26.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:26.355 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:26.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:24.967814+0000 mon.smithi084 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.106:0/252596859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:24.923541+0000 mgr.smithi084.sihbqg (mgr.14162) 91 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- 172.21.15.106:0/1454493307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 msgr2=0x7fad140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 --2- 172.21.15.106:0/1454493307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f90f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fad08009fb0 tx=0x7fad08009b30).stop 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- 172.21.15.106:0/1454493307 shutdown_connections 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 --2- 172.21.15.106:0/1454493307 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f90f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- 172.21.15.106:0/1454493307 >> 172.21.15.106:0/1454493307 conn(0x7fad140f0b70 msgr2=0x7fad140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- 172.21.15.106:0/1454493307 shutdown_connections 2024-01-18T07:22:27.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- 172.21.15.106:0/1454493307 wait complete. 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 Processor -- start 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.496+0000 7fad1930a700 1 -- start start 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad1930a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad1930a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140f4ff0 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad12ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad12ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50334/0 (socket says 172.21.15.106:50334) 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad12ffd700 1 -- 172.21.15.106:0/591652094 learned_addr learned my addr 172.21.15.106:0/591652094 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad12ffd700 1 -- 172.21.15.106:0/591652094 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad08005040 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad12ffd700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fad08013510 tx=0x7fad0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad08009320 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad140f5190 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad08013f00 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.497+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad08014dc0 con 0x7fad14060280 2024-01-18T07:22:27.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.498+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad140f5620 con 0x7fad14060280 2024-01-18T07:22:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.498+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad080095f0 con 0x7fad14060280 2024-01-18T07:22:27.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.498+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14059100 con 0x7fad14060280 2024-01-18T07:22:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.498+0000 7facfb7fe700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 0x7facfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.498+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad08041000 con 0x7fad14060280 2024-01-18T07:22:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.499+0000 7fad127fc700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 0x7facfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:27.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.499+0000 7fad127fc700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 0x7facfc032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fad040096a0 tx=0x7fad04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:27.503 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.503+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad08014420 con 0x7fad14060280 2024-01-18T07:22:27.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.708+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad140fbf50 con 0x7fad14060280 2024-01-18T07:22:27.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.708+0000 7facfb7fe700 1 -- 172.21.15.106:0/591652094 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad08010230 con 0x7fad14060280 2024-01-18T07:22:27.709 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:27.709 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 msgr2=0x7facfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 0x7facfc032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fad040096a0 tx=0x7fad04006b40).stop 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 msgr2=0x7fad140f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fad08013510 tx=0x7fad0800b040).stop 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 shutdown_connections 2024-01-18T07:22:27.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7facfc02ff60 0x7facfc032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:27.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 --2- 172.21.15.106:0/591652094 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fad14060280 0x7fad140f6b70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:27.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 >> 172.21.15.106:0/591652094 conn(0x7fad140f0b70 msgr2=0x7fad140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:27.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 shutdown_connections 2024-01-18T07:22:27.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:27.710+0000 7fad1930a700 1 -- 172.21.15.106:0/591652094 wait complete. 2024-01-18T07:22:27.711 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:27.708897+0000 mon.smithi084 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.106:0/591652094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:29.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:29.078 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:26.923836+0000 mgr.smithi084.sihbqg (mgr.14162) 92 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:30.204 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- 172.21.15.106:0/4075422195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 msgr2=0x7f9b1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 --2- 172.21.15.106:0/4075422195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9b10009fb0 tx=0x7f9b10009b30).stop 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- 172.21.15.106:0/4075422195 shutdown_connections 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 --2- 172.21.15.106:0/4075422195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- 172.21.15.106:0/4075422195 >> 172.21.15.106:0/4075422195 conn(0x7f9b1c0f1c70 msgr2=0x7f9b1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- 172.21.15.106:0/4075422195 shutdown_connections 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- 172.21.15.106:0/4075422195 wait complete. 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 Processor -- start 2024-01-18T07:22:30.205 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.204+0000 7f9b22f17700 1 -- start start 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b22f17700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b22f17700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c106c20 con 0x7f9b1c0f6510 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b20cb3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b20cb3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50358/0 (socket says 172.21.15.106:50358) 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b20cb3700 1 -- 172.21.15.106:0/1838784820 learned_addr learned my addr 172.21.15.106:0/1838784820 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b20cb3700 1 -- 172.21.15.106:0/1838784820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10005040 con 0x7f9b1c0f6510 2024-01-18T07:22:30.207 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.205+0000 7f9b20cb3700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9b10013510 tx=0x7f9b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b10009320 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b10013f00 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1c106e20 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b10014d90 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c1072b0 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b10014420 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.206+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1c045ad0 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.207+0000 7f9b197fa700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 0x7f9b0c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.207+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b10041090 con 0x7f9b1c0f6510 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.207+0000 7f9b1bfff700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 0x7f9b0c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:30.208 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.207+0000 7f9b1bfff700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 0x7f9b0c0324c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:30.211 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.211+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b1006d2b0 con 0x7f9b1c0f6510 2024-01-18T07:22:30.417 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.417+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b1c0fbf60 con 0x7f9b1c0f6510 2024-01-18T07:22:30.418 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.418+0000 7f9b197fa700 1 -- 172.21.15.106:0/1838784820 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b100095f0 con 0x7f9b1c0f6510 2024-01-18T07:22:30.418 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:30.418 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 msgr2=0x7f9b0c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 0x7f9b0c0324c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).stop 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 msgr2=0x7f9b1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f9b10013510 tx=0x7f9b1000b040).stop 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 shutdown_connections 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9b0c030000 0x7f9b0c0324c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 --2- 172.21.15.106:0/1838784820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9b1c0f6510 0x7f9b1c106730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 >> 172.21.15.106:0/1838784820 conn(0x7f9b1c0f1c70 msgr2=0x7f9b1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 shutdown_connections 2024-01-18T07:22:30.420 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:30.419+0000 7f9b22f17700 1 -- 172.21.15.106:0/1838784820 wait complete. 2024-01-18T07:22:30.421 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:28.924136+0000 mgr.smithi084.sihbqg (mgr.14162) 93 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:30.418237+0000 mon.smithi084 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.106:0/1838784820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:31.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:31.841 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:32.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 -- 172.21.15.106:0/1701648957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 msgr2=0x7f44900f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:32.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 --2- 172.21.15.106:0/1701648957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44900f66c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4484009fb0 tx=0x7f4484009b30).stop 2024-01-18T07:22:32.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 -- 172.21.15.106:0/1701648957 shutdown_connections 2024-01-18T07:22:32.995 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 --2- 172.21.15.106:0/1701648957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44900f66c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 -- 172.21.15.106:0/1701648957 >> 172.21.15.106:0/1701648957 conn(0x7f44900f1aa0 msgr2=0x7f44900f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.993+0000 7f4496d4e700 1 -- 172.21.15.106:0/1701648957 shutdown_connections 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4496d4e700 1 -- 172.21.15.106:0/1701648957 wait complete. 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4496d4e700 1 Processor -- start 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4496d4e700 1 -- start start 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4496d4e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4496d4e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44901069a0 con 0x7f44900f62a0 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4495d4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4495d4c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:50386/0 (socket says 172.21.15.106:50386) 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.994+0000 7f4495d4c700 1 -- 172.21.15.106:0/322146770 learned_addr learned my addr 172.21.15.106:0/322146770 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f4495d4c700 1 -- 172.21.15.106:0/322146770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484005040 con 0x7f44900f62a0 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f4495d4c700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4484013510 tx=0x7f448400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:32.996 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4484009320 con 0x7f44900f62a0 2024-01-18T07:22:32.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4490106ba0 con 0x7f44900f62a0 2024-01-18T07:22:32.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4484013f00 con 0x7f44900f62a0 2024-01-18T07:22:32.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4484014dc0 con 0x7f44900f62a0 2024-01-18T07:22:32.997 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.995+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4490107030 con 0x7f44900f62a0 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.996+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4484014420 con 0x7f44900f62a0 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.996+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4490045ad0 con 0x7f44900f62a0 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.996+0000 7f44827fc700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 0x7f447c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.996+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4484041110 con 0x7f44900f62a0 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.996+0000 7f449554b700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 0x7f447c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:32.998 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:32.997+0000 7f449554b700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 0x7f447c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f448c0096a0 tx=0x7f448c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:33.001 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.001+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f448401cb10 con 0x7f44900f62a0 2024-01-18T07:22:33.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.209+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4490107ee0 con 0x7f44900f62a0 2024-01-18T07:22:33.210 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.210+0000 7f44827fc700 1 -- 172.21.15.106:0/322146770 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f448400f750 con 0x7f44900f62a0 2024-01-18T07:22:33.210 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:33.210 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 msgr2=0x7f447c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 0x7f447c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f448c0096a0 tx=0x7f448c006b40).stop 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 msgr2=0x7f44901064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f4484013510 tx=0x7f448400b040).stop 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 shutdown_connections 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f447c02ff60 0x7f447c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 --2- 172.21.15.106:0/322146770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f44900f62a0 0x7f44901064b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 >> 172.21.15.106:0/322146770 conn(0x7f44900f1aa0 msgr2=0x7f44900f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:33.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 shutdown_connections 2024-01-18T07:22:33.213 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:33.212+0000 7f4496d4e700 1 -- 172.21.15.106:0/322146770 wait complete. 2024-01-18T07:22:33.213 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:30.924507+0000 mgr.smithi084.sihbqg (mgr.14162) 94 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:33.210702+0000 mon.smithi084 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.106:0/322146770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:34.599 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:34.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:32.924883+0000 mgr.smithi084.sihbqg (mgr.14162) 95 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:35.707 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 -- 172.21.15.106:0/568699184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 msgr2=0x7f69e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 --2- 172.21.15.106:0/568699184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e40f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f69d4009fb0 tx=0x7f69d4009b30).stop 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 -- 172.21.15.106:0/568699184 shutdown_connections 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 --2- 172.21.15.106:0/568699184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e40f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 -- 172.21.15.106:0/568699184 >> 172.21.15.106:0/568699184 conn(0x7f69e40f0b70 msgr2=0x7f69e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 -- 172.21.15.106:0/568699184 shutdown_connections 2024-01-18T07:22:35.708 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.707+0000 7f69ea55c700 1 -- 172.21.15.106:0/568699184 wait complete. 2024-01-18T07:22:35.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69ea55c700 1 Processor -- start 2024-01-18T07:22:35.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69ea55c700 1 -- start start 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69ea55c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69ea55c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4060240 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69e3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69e3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55194/0 (socket says 172.21.15.106:55194) 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.708+0000 7f69e3fff700 1 -- 172.21.15.106:0/1245321428 learned_addr learned my addr 172.21.15.106:0/1245321428 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69e3fff700 1 -- 172.21.15.106:0/1245321428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4005040 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69e3fff700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f69d4013510 tx=0x7f69d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4009320 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f69d4013f00 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e4060440 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4014d90 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.709+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4109ae0 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.710+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69d40095f0 con 0x7f69e40f5410 2024-01-18T07:22:35.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.710+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4045ad0 con 0x7f69e40f5410 2024-01-18T07:22:35.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.710+0000 7f69e0ff9700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 0x7f69cc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:35.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.710+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69d40410a0 con 0x7f69e40f5410 2024-01-18T07:22:35.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.710+0000 7f69e37fe700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 0x7f69cc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:35.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.711+0000 7f69e37fe700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 0x7f69cc032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f69d80096a0 tx=0x7f69d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:35.714 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.714+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69d406d2b0 con 0x7f69e40f5410 2024-01-18T07:22:35.920 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.920+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69e40fae60 con 0x7f69e40f5410 2024-01-18T07:22:35.921 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.921+0000 7f69e0ff9700 1 -- 172.21.15.106:0/1245321428 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69d4014420 con 0x7f69e40f5410 2024-01-18T07:22:35.921 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:35.921 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:35.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 msgr2=0x7f69cc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:35.922 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 0x7f69cc032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f69d80096a0 tx=0x7f69d8006b40).stop 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 msgr2=0x7f69e4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f69d4013510 tx=0x7f69d400b040).stop 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 shutdown_connections 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f69cc02ffb0 0x7f69cc032470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 --2- 172.21.15.106:0/1245321428 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f69e40f5410 0x7f69e4109430 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 >> 172.21.15.106:0/1245321428 conn(0x7f69e40f0b70 msgr2=0x7f69e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 shutdown_connections 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:35.922+0000 7f69ea55c700 1 -- 172.21.15.106:0/1245321428 wait complete. 2024-01-18T07:22:35.923 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:35.033922+0000 mon.smithi084 (mon.0) 185 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:22:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:35.041365+0000 mon.smithi084 (mon.0) 186 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:22:37.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:37.315 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:34.925211+0000 mgr.smithi084.sihbqg (mgr.14162) 96 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:35.921077+0000 mon.smithi084 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.106:0/1245321428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:36.925583+0000 mgr.smithi084.sihbqg (mgr.14162) 97 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:38.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 -- 172.21.15.106:0/4020355571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 msgr2=0x7fc5780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:38.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/4020355571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc5780f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fc568009fb0 tx=0x7fc568009b30).stop 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 -- 172.21.15.106:0/4020355571 shutdown_connections 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/4020355571 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc5780f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 -- 172.21.15.106:0/4020355571 >> 172.21.15.106:0/4020355571 conn(0x7fc5780f1c70 msgr2=0x7fc5780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 -- 172.21.15.106:0/4020355571 shutdown_connections 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.451+0000 7fc57f3fd700 1 -- 172.21.15.106:0/4020355571 wait complete. 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57f3fd700 1 Processor -- start 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57f3fd700 1 -- start start 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57f3fd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57f3fd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc578108e00 con 0x7fc5780f6510 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57d199700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.452+0000 7fc57d199700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55222/0 (socket says 172.21.15.106:55222) 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc57d199700 1 -- 172.21.15.106:0/3166178239 learned_addr learned my addr 172.21.15.106:0/3166178239 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:38.455 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc57d199700 1 -- 172.21.15.106:0/3166178239 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc568005040 con 0x7fc5780f6510 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc57d199700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc568013510 tx=0x7fc56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc568009320 con 0x7fc5780f6510 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc568013f00 con 0x7fc5780f6510 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc578109000 con 0x7fc5780f6510 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc568014d90 con 0x7fc5780f6510 2024-01-18T07:22:38.456 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.453+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc578109490 con 0x7fc5780f6510 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.454+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc568014420 con 0x7fc5780f6510 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.454+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc578045ad0 con 0x7fc5780f6510 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.454+0000 7fc56dffb700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 0x7fc564032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.454+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc568041000 con 0x7fc5780f6510 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.455+0000 7fc57c998700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 0x7fc564032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:38.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.455+0000 7fc57c998700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 0x7fc564032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc5740096a0 tx=0x7fc574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:38.460 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.459+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc568026bf0 con 0x7fc5780f6510 2024-01-18T07:22:38.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.666+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc57810a340 con 0x7fc5780f6510 2024-01-18T07:22:38.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.667+0000 7fc56dffb700 1 -- 172.21.15.106:0/3166178239 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc56801db70 con 0x7fc5780f6510 2024-01-18T07:22:38.668 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:38.668 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 msgr2=0x7fc564032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 0x7fc564032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc5740096a0 tx=0x7fc574006b40).stop 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 msgr2=0x7fc578108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fc568013510 tx=0x7fc56800b040).stop 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 shutdown_connections 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc56402ff60 0x7fc564032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 --2- 172.21.15.106:0/3166178239 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc5780f6510 0x7fc578108910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 >> 172.21.15.106:0/3166178239 conn(0x7fc5780f1c70 msgr2=0x7fc5780f26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:38.670 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 shutdown_connections 2024-01-18T07:22:38.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:38.669+0000 7fc57f3fd700 1 -- 172.21.15.106:0/3166178239 wait complete. 2024-01-18T07:22:38.671 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:38.668371+0000 mon.smithi084 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.106:0/3166178239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:40.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:40.074 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:38.925966+0000 mgr.smithi084.sihbqg (mgr.14162) 98 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.268+0000 7fa4cb59e700 1 -- 172.21.15.106:0/2373285681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 msgr2=0x7fa4cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.268+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/2373285681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc0f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa4c0009fb0 tx=0x7fa4c0009b30).stop 2024-01-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- 172.21.15.106:0/2373285681 shutdown_connections 2024-01-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/2373285681 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc0f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:41.271 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- 172.21.15.106:0/2373285681 >> 172.21.15.106:0/2373285681 conn(0x7fa4cc0f09a0 msgr2=0x7fa4cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- 172.21.15.106:0/2373285681 shutdown_connections 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- 172.21.15.106:0/2373285681 wait complete. 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 Processor -- start 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- start start 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4cb59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc108bc0 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4ca59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4ca59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55240/0 (socket says 172.21.15.106:55240) 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.269+0000 7fa4ca59c700 1 -- 172.21.15.106:0/732732819 learned_addr learned my addr 172.21.15.106:0/732732819 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4ca59c700 1 -- 172.21.15.106:0/732732819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0005040 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4ca59c700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa4c0013510 tx=0x7fa4c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:41.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c0009320 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4cc108dc0 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4c0013f00 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4c0014dc0 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.270+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4cc109250 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.271+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4c00095f0 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.271+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc045ad0 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.271+0000 7fa4baffd700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 0x7fa4b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.271+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4c0041040 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.271+0000 7fa4c9d9b700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 0x7fa4b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:41.274 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.272+0000 7fa4c9d9b700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 0x7fa4b4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa4bc0096a0 tx=0x7fa4bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:41.276 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.275+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4c0014420 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.480 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.479+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4cc10a100 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.481 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.480+0000 7fa4baffd700 1 -- 172.21.15.106:0/732732819 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4c0010230 con 0x7fa4cc0f51a0 2024-01-18T07:22:41.481 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:41.481 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 msgr2=0x7fa4b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 0x7fa4b4032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa4bc0096a0 tx=0x7fa4bc006b40).stop 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 msgr2=0x7fa4cc1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa4c0013510 tx=0x7fa4c000b040).stop 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 shutdown_connections 2024-01-18T07:22:41.483 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa4b402ff60 0x7fa4b4032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:41.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 --2- 172.21.15.106:0/732732819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4cc0f51a0 0x7fa4cc1086d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:41.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 >> 172.21.15.106:0/732732819 conn(0x7fa4cc0f09a0 msgr2=0x7fa4cc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:41.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 shutdown_connections 2024-01-18T07:22:41.484 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:41.482+0000 7fa4cb59e700 1 -- 172.21.15.106:0/732732819 wait complete. 2024-01-18T07:22:41.484 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:41.481707+0000 mon.smithi084 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.106:0/732732819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:42.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:42.876 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:40.926222+0000 mgr.smithi084.sihbqg (mgr.14162) 99 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 -- 172.21.15.106:0/4235281152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 msgr2=0x7f5abc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 --2- 172.21.15.106:0/4235281152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc0f9150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f5aac009fb0 tx=0x7f5aac009b30).stop 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 -- 172.21.15.106:0/4235281152 shutdown_connections 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 --2- 172.21.15.106:0/4235281152 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc0f9150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 -- 172.21.15.106:0/4235281152 >> 172.21.15.106:0/4235281152 conn(0x7f5abc0f0b70 msgr2=0x7f5abc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 -- 172.21.15.106:0/4235281152 shutdown_connections 2024-01-18T07:22:44.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.026+0000 7f5ac3696700 1 -- 172.21.15.106:0/4235281152 wait complete. 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac3696700 1 Processor -- start 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac3696700 1 -- start start 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac3696700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac3696700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc106bd0 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac1432700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac1432700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:55262/0 (socket says 172.21.15.106:55262) 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac1432700 1 -- 172.21.15.106:0/856134987 learned_addr learned my addr 172.21.15.106:0/856134987 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.027+0000 7f5ac1432700 1 -- 172.21.15.106:0/856134987 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aac005040 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ac1432700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5aac013510 tx=0x7f5aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aac009320 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc106dd0 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5aac013f00 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aac014dc0 con 0x7f5abc060220 2024-01-18T07:22:44.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.028+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc107260 con 0x7f5abc060220 2024-01-18T07:22:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.029+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5aac0095f0 con 0x7f5abc060220 2024-01-18T07:22:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.029+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc045ad0 con 0x7f5abc060220 2024-01-18T07:22:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.029+0000 7f5ab1ffb700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 0x7f5aa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:44.031 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.029+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5aac0410e0 con 0x7f5abc060220 2024-01-18T07:22:44.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.029+0000 7f5ac0c31700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 0x7f5aa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:44.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.030+0000 7f5ac0c31700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 0x7f5aa8032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5ab80096a0 tx=0x7f5ab8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:44.034 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.033+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5aac014420 con 0x7f5abc060220 2024-01-18T07:22:44.240 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.239+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5abc1080b0 con 0x7f5abc060220 2024-01-18T07:22:44.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.240+0000 7f5ab1ffb700 1 -- 172.21.15.106:0/856134987 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5aac01c920 con 0x7f5abc060220 2024-01-18T07:22:44.242 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:44.242 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 msgr2=0x7f5aa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 0x7f5aa8032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5ab80096a0 tx=0x7f5ab8006b40).stop 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 msgr2=0x7f5abc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5aac013510 tx=0x7f5aac00b040).stop 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 shutdown_connections 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f5aa802ff60 0x7f5aa8032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 --2- 172.21.15.106:0/856134987 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f5abc060220 0x7f5abc1066e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 >> 172.21.15.106:0/856134987 conn(0x7f5abc0f0b70 msgr2=0x7f5abc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:44.243 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 shutdown_connections 2024-01-18T07:22:44.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:44.242+0000 7f5ac3696700 1 -- 172.21.15.106:0/856134987 wait complete. 2024-01-18T07:22:44.244 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:42.926523+0000 mgr.smithi084.sihbqg (mgr.14162) 100 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:45.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:44.241546+0000 mon.smithi084 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.106:0/856134987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:45.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:45.622 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:46.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- 172.21.15.106:0/3680578778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 msgr2=0x7f21700f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 --2- 172.21.15.106:0/3680578778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21700f7d00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f2160009fb0 tx=0x7f2160009b30).stop 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- 172.21.15.106:0/3680578778 shutdown_connections 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 --2- 172.21.15.106:0/3680578778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21700f7d00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- 172.21.15.106:0/3680578778 >> 172.21.15.106:0/3680578778 conn(0x7f21700f0120 msgr2=0x7f21700f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- 172.21.15.106:0/3680578778 shutdown_connections 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- 172.21.15.106:0/3680578778 wait complete. 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 Processor -- start 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.722+0000 7f216f59e700 1 -- start start 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216f59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:46.724 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216f59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21701069d0 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:34954/0 (socket says 172.21.15.106:34954) 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216e59c700 1 -- 172.21.15.106:0/794674383 learned_addr learned my addr 172.21.15.106:0/794674383 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216e59c700 1 -- 172.21.15.106:0/794674383 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2160005040 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216e59c700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2160013510 tx=0x7f216000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2160009320 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2170106bd0 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2160013f00 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2160014dc0 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.723+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2170107060 con 0x7f21700f78e0 2024-01-18T07:22:46.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.724+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2160014420 con 0x7f21700f78e0 2024-01-18T07:22:46.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.724+0000 7f215effd700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:46.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.724+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2170045ad0 con 0x7f21700f78e0 2024-01-18T07:22:46.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.724+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2160041110 con 0x7f21700f78e0 2024-01-18T07:22:46.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.724+0000 7f216dd9b700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:46.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.725+0000 7f216dd9b700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 0x7f2158032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:46.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.729+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21600095f0 con 0x7f21700f78e0 2024-01-18T07:22:46.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.936+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2170107f10 con 0x7f21700f78e0 2024-01-18T07:22:46.937 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.936+0000 7f215effd700 1 -- 172.21.15.106:0/794674383 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2160015040 con 0x7f21700f78e0 2024-01-18T07:22:46.938 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:46.938 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:46.939 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 msgr2=0x7f2158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 0x7f2158032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).stop 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 msgr2=0x7f21701064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f2160013510 tx=0x7f216000b040).stop 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 shutdown_connections 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 --2- 172.21.15.106:0/794674383 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f21700f78e0 0x7f21701064e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 >> 172.21.15.106:0/794674383 conn(0x7f21700f0120 msgr2=0x7f21700f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 shutdown_connections 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:46.938+0000 7f216f59e700 1 -- 172.21.15.106:0/794674383 wait complete. 2024-01-18T07:22:46.940 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:44.926896+0000 mgr.smithi084.sihbqg (mgr.14162) 101 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:48.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:48.338 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:48.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:46.938104+0000 mon.smithi084 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.106:0/794674383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:46.927301+0000 mgr.smithi084.sihbqg (mgr.14162) 102 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 -- 172.21.15.106:0/2259116269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 msgr2=0x7f91680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 --2- 172.21.15.106:0/2259116269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f91680f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f915c009fb0 tx=0x7f915c009b30).stop 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 -- 172.21.15.106:0/2259116269 shutdown_connections 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 --2- 172.21.15.106:0/2259116269 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f91680f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 -- 172.21.15.106:0/2259116269 >> 172.21.15.106:0/2259116269 conn(0x7f91680f1c70 msgr2=0x7f91680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 -- 172.21.15.106:0/2259116269 shutdown_connections 2024-01-18T07:22:49.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.495+0000 7f916d299700 1 -- 172.21.15.106:0/2259116269 wait complete. 2024-01-18T07:22:49.498 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.496+0000 7f916d299700 1 Processor -- start 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.496+0000 7f916d299700 1 -- start start 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.496+0000 7f916d299700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.496+0000 7f916d299700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168106c20 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:34980/0 (socket says 172.21.15.106:34980) 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f9166ffd700 1 -- 172.21.15.106:0/2152053091 learned_addr learned my addr 172.21.15.106:0/2152053091 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f9166ffd700 1 -- 172.21.15.106:0/2152053091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c005040 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f9166ffd700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f915c013510 tx=0x7f915c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915c009320 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9168106e20 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f915c013f00 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915c014d90 con 0x7f91680f6510 2024-01-18T07:22:49.499 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.497+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91681072b0 con 0x7f91680f6510 2024-01-18T07:22:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.498+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f915c0095f0 con 0x7f91680f6510 2024-01-18T07:22:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.498+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9168045ad0 con 0x7f91680f6510 2024-01-18T07:22:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.498+0000 7f914f7fe700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 0x7f9150032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:49.500 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.498+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f915c041080 con 0x7f91680f6510 2024-01-18T07:22:49.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.499+0000 7f91667fc700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 0x7f9150032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:49.501 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.499+0000 7f91667fc700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 0x7f9150032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f91580096a0 tx=0x7f9158006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:49.504 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.503+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f915c014420 con 0x7f91680f6510 2024-01-18T07:22:49.709 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.708+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91680fbf60 con 0x7f91680f6510 2024-01-18T07:22:49.710 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.708+0000 7f914f7fe700 1 -- 172.21.15.106:0/2152053091 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f915c010230 con 0x7f91680f6510 2024-01-18T07:22:49.710 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:49.710 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:49.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 msgr2=0x7f9150032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:49.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 0x7f9150032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f91580096a0 tx=0x7f9158006b40).stop 2024-01-18T07:22:49.711 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 msgr2=0x7f9168106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f915c013510 tx=0x7f915c00b040).stop 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 shutdown_connections 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f915002ff60 0x7f9150032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 --2- 172.21.15.106:0/2152053091 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f91680f6510 0x7f9168106730 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 >> 172.21.15.106:0/2152053091 conn(0x7f91680f1c70 msgr2=0x7f91680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 shutdown_connections 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:49.710+0000 7f916d299700 1 -- 172.21.15.106:0/2152053091 wait complete. 2024-01-18T07:22:49.712 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:49.710060+0000 mon.smithi084 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.106:0/2152053091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:51.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:51.092 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:48.927601+0000 mgr.smithi084.sihbqg (mgr.14162) 103 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 -- 172.21.15.106:0/3659460817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 msgr2=0x7fc1500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 --2- 172.21.15.106:0/3659460817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc1500f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fc14c009fb0 tx=0x7fc14c009b30).stop 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 -- 172.21.15.106:0/3659460817 shutdown_connections 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 --2- 172.21.15.106:0/3659460817 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc1500f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 -- 172.21.15.106:0/3659460817 >> 172.21.15.106:0/3659460817 conn(0x7fc1500f0b70 msgr2=0x7fc1500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 -- 172.21.15.106:0/3659460817 shutdown_connections 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 -- 172.21.15.106:0/3659460817 wait complete. 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.279+0000 7fc15767f700 1 Processor -- start 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15767f700 1 -- start start 2024-01-18T07:22:52.282 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15767f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15767f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc150060240 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15541b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15541b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35000/0 (socket says 172.21.15.106:35000) 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15541b700 1 -- 172.21.15.106:0/1384143302 learned_addr learned my addr 172.21.15.106:0/1384143302 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15541b700 1 -- 172.21.15.106:0/1384143302 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c005040 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc15541b700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc14c013510 tx=0x7fc14c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.280+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c009320 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.281+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc150060440 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.281+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc14c013f00 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.281+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc14c014d90 con 0x7fc1500f5410 2024-01-18T07:22:52.283 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.281+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc150109ae0 con 0x7fc1500f5410 2024-01-18T07:22:52.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.281+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc150045ad0 con 0x7fc1500f5410 2024-01-18T07:22:52.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.282+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc14c0095f0 con 0x7fc1500f5410 2024-01-18T07:22:52.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.282+0000 7fc145ffb700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 0x7fc13c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:52.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.282+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc14c040fe0 con 0x7fc1500f5410 2024-01-18T07:22:52.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.282+0000 7fc154c1a700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 0x7fc13c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:52.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.282+0000 7fc154c1a700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 0x7fc13c032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc1400096a0 tx=0x7fc140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:52.288 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.286+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc14c06d2b0 con 0x7fc1500f5410 2024-01-18T07:22:52.493 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.492+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1500fae60 con 0x7fc1500f5410 2024-01-18T07:22:52.494 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.493+0000 7fc145ffb700 1 -- 172.21.15.106:0/1384143302 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc14c014420 con 0x7fc1500f5410 2024-01-18T07:22:52.494 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:52.494 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:52.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 msgr2=0x7fc13c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:52.496 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 0x7fc13c032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fc1400096a0 tx=0x7fc140006b40).stop 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 msgr2=0x7fc150109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc14c013510 tx=0x7fc14c00b040).stop 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 shutdown_connections 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc13c02ffb0 0x7fc13c032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 --2- 172.21.15.106:0/1384143302 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1500f5410 0x7fc150109430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 >> 172.21.15.106:0/1384143302 conn(0x7fc1500f0b70 msgr2=0x7fc1500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 shutdown_connections 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:52.495+0000 7fc15767f700 1 -- 172.21.15.106:0/1384143302 wait complete. 2024-01-18T07:22:52.497 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:50.927963+0000 mgr.smithi084.sihbqg (mgr.14162) 104 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:52.494758+0000 mon.smithi084 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.106:0/1384143302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:53.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:53.884 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3423971581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 msgr2=0x7f07e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3423971581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f07d4009fb0 tx=0x7f07d4009b30).stop 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3423971581 shutdown_connections 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3423971581 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3423971581 >> 172.21.15.106:0/3423971581 conn(0x7f07e40f0b70 msgr2=0x7f07e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:55.029 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3423971581 shutdown_connections 2024-01-18T07:22:55.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.027+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3423971581 wait complete. 2024-01-18T07:22:55.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07ea5c9700 1 Processor -- start 2024-01-18T07:22:55.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07ea5c9700 1 -- start start 2024-01-18T07:22:55.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07ea5c9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:55.030 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07ea5c9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e40fb820 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07e3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07e3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:35018/0 (socket says 172.21.15.106:35018) 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.028+0000 7f07e3fff700 1 -- 172.21.15.106:0/3346761256 learned_addr learned my addr 172.21.15.106:0/3346761256 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07e3fff700 1 -- 172.21.15.106:0/3346761256 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d4005040 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07e3fff700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f07d4013510 tx=0x7f07d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07d4009320 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e40fb9c0 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07d4013f00 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e40fbe50 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.029+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07d4014d90 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.030+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07d4014420 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.030+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c8004fa0 con 0x7f07e40f5410 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.030+0000 7f07e0ff9700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 0x7f07d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:55.032 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.030+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07d4040f50 con 0x7f07e40f5410 2024-01-18T07:22:55.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.030+0000 7f07e37fe700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 0x7f07d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:55.033 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.031+0000 7f07e37fe700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 0x7f07d0032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:55.036 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.035+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07d406d2b0 con 0x7f07e40f5410 2024-01-18T07:22:55.241 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.240+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07c8005670 con 0x7f07e40f5410 2024-01-18T07:22:55.242 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.241+0000 7f07e0ff9700 1 -- 172.21.15.106:0/3346761256 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07d40095f0 con 0x7f07e40f5410 2024-01-18T07:22:55.242 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:55.242 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.242+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 msgr2=0x7f07d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.242+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 0x7f07d0032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f07cc0096a0 tx=0x7f07cc006b40).stop 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.242+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 msgr2=0x7f07e40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.242+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f07d4013510 tx=0x7f07d400b040).stop 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 shutdown_connections 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f07d002ff60 0x7f07d0032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 --2- 172.21.15.106:0/3346761256 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07e40f5410 0x7f07e40fd3a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 >> 172.21.15.106:0/3346761256 conn(0x7f07e40f0b70 msgr2=0x7f07e405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:55.244 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 shutdown_connections 2024-01-18T07:22:55.245 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:55.243+0000 7f07ea5c9700 1 -- 172.21.15.106:0/3346761256 wait complete. 2024-01-18T07:22:55.245 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:52.928265+0000 mgr.smithi084.sihbqg (mgr.14162) 105 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:56.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:55.242382+0000 mon.smithi084 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.106:0/3346761256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:56.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:56.624 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:22:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:54.928555+0000 mgr.smithi084.sihbqg (mgr.14162) 106 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/18470819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 msgr2=0x7fb8f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/18470819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f80f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb8e8009fb0 tx=0x7fb8e8009b30).stop 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/18470819 shutdown_connections 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/18470819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f80f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/18470819 >> 172.21.15.106:0/18470819 conn(0x7fb8f80f0b70 msgr2=0x7fb8f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/18470819 shutdown_connections 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/18470819 wait complete. 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.715+0000 7fb8ffdd6700 1 Processor -- start 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8ffdd6700 1 -- start start 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8ffdd6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:57.718 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8ffdd6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f8107210 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8fdb72700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8fdb72700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51466/0 (socket says 172.21.15.106:51466) 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8fdb72700 1 -- 172.21.15.106:0/2880319409 learned_addr learned my addr 172.21.15.106:0/2880319409 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8fdb72700 1 -- 172.21.15.106:0/2880319409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8005040 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8fdb72700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb8e8013510 tx=0x7fb8e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e8009320 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.716+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f8060220 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8e8013f00 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e8014d90 con 0x7fb8f80f5410 2024-01-18T07:22:57.719 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f810bd10 con 0x7fb8f80f5410 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8e80095f0 con 0x7fb8f80f5410 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f8045ad0 con 0x7fb8f80f5410 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.717+0000 7fb8ee7fc700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 0x7fb8e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.718+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8e80410c0 con 0x7fb8f80f5410 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.718+0000 7fb8fd371700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 0x7fb8e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:22:57.720 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.718+0000 7fb8fd371700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 0x7fb8e4032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb8f40096a0 tx=0x7fb8f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:22:57.723 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.722+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e8014420 con 0x7fb8f80f5410 2024-01-18T07:22:57.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.927+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8f80fae60 con 0x7fb8f80f5410 2024-01-18T07:22:57.929 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.928+0000 7fb8ee7fc700 1 -- 172.21.15.106:0/2880319409 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8e801d7f0 con 0x7fb8f80f5410 2024-01-18T07:22:57.929 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:22:57.929 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 msgr2=0x7fb8e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 0x7fb8e4032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb8f40096a0 tx=0x7fb8f4006b40).stop 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 msgr2=0x7fb8f810b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fb8e8013510 tx=0x7fb8e800b040).stop 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 shutdown_connections 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb8e402ff60 0x7fb8e4032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 --2- 172.21.15.106:0/2880319409 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb8f80f5410 0x7fb8f810b820 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 >> 172.21.15.106:0/2880319409 conn(0x7fb8f80f0b70 msgr2=0x7fb8f80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 shutdown_connections 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:22:57.929+0000 7fb8ffdd6700 1 -- 172.21.15.106:0/2880319409 wait complete. 2024-01-18T07:22:57.931 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:22:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:56.928920+0000 mgr.smithi084.sihbqg (mgr.14162) 107 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:22:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:22:57.929253+0000 mon.smithi084 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.106:0/2880319409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:22:59.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:22:59.325 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:22:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:22:58.929188+0000 mgr.smithi084.sihbqg (mgr.14162) 108 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- 172.21.15.106:0/3119371306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 msgr2=0x7fac280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 --2- 172.21.15.106:0/3119371306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac280f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fac18009fb0 tx=0x7fac18009b30).stop 2024-01-18T07:23:00.451 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- 172.21.15.106:0/3119371306 shutdown_connections 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 --2- 172.21.15.106:0/3119371306 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac280f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- 172.21.15.106:0/3119371306 >> 172.21.15.106:0/3119371306 conn(0x7fac280f09a0 msgr2=0x7fac280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- 172.21.15.106:0/3119371306 shutdown_connections 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- 172.21.15.106:0/3119371306 wait complete. 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 Processor -- start 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- start start 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.450+0000 7fac2759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac28060240 con 0x7fac280f51a0 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51490/0 (socket says 172.21.15.106:51490) 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2659c700 1 -- 172.21.15.106:0/2539269818 learned_addr learned my addr 172.21.15.106:0/2539269818 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2659c700 1 -- 172.21.15.106:0/2539269818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18005040 con 0x7fac280f51a0 2024-01-18T07:23:00.452 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2659c700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fac18013510 tx=0x7fac1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:00.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac18009320 con 0x7fac280f51a0 2024-01-18T07:23:00.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac28060440 con 0x7fac280f51a0 2024-01-18T07:23:00.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac18013f00 con 0x7fac280f51a0 2024-01-18T07:23:00.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac18014dc0 con 0x7fac280f51a0 2024-01-18T07:23:00.453 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.451+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac28109890 con 0x7fac280f51a0 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.452+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac1801d3d0 con 0x7fac280f51a0 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.452+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac28045ad0 con 0x7fac280f51a0 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.452+0000 7fac16ffd700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 0x7fac10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.452+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac180413a0 con 0x7fac280f51a0 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.453+0000 7fac25d9b700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 0x7fac10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:00.454 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.453+0000 7fac25d9b700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 0x7fac10032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fac1c0096a0 tx=0x7fac1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:00.457 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.457+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac1801cac0 con 0x7fac280f51a0 2024-01-18T07:23:00.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.664+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac2810a0b0 con 0x7fac280f51a0 2024-01-18T07:23:00.665 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.665+0000 7fac16ffd700 1 -- 172.21.15.106:0/2539269818 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac1801cac0 con 0x7fac280f51a0 2024-01-18T07:23:00.666 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:00.666 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 msgr2=0x7fac10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 0x7fac10032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fac1c0096a0 tx=0x7fac1c006b40).stop 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 msgr2=0x7fac281091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fac18013510 tx=0x7fac1800b040).stop 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 shutdown_connections 2024-01-18T07:23:00.667 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fac1002ff60 0x7fac10032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 --2- 172.21.15.106:0/2539269818 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fac280f51a0 0x7fac281091e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 >> 172.21.15.106:0/2539269818 conn(0x7fac280f09a0 msgr2=0x7fac280f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 shutdown_connections 2024-01-18T07:23:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:00.667+0000 7fac2759e700 1 -- 172.21.15.106:0/2539269818 wait complete. 2024-01-18T07:23:00.668 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:00.666074+0000 mon.smithi084 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.106:0/2539269818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:02.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:02.069 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:03.214 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.212+0000 7f232acc2700 1 -- 172.21.15.106:0/1987519567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 msgr2=0x7f23240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.212+0000 7f232acc2700 1 --2- 172.21.15.106:0/1987519567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f23240f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2318009fb0 tx=0x7f2318009b30).stop 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- 172.21.15.106:0/1987519567 shutdown_connections 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 --2- 172.21.15.106:0/1987519567 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f23240f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- 172.21.15.106:0/1987519567 >> 172.21.15.106:0/1987519567 conn(0x7f23240f1c70 msgr2=0x7f23240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- 172.21.15.106:0/1987519567 shutdown_connections 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- 172.21.15.106:0/1987519567 wait complete. 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 Processor -- start 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- start start 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.213+0000 7f232acc2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324069e20 con 0x7f23240f6510 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f2328a5e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f2328a5e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51512/0 (socket says 172.21.15.106:51512) 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f2328a5e700 1 -- 172.21.15.106:0/506622986 learned_addr learned my addr 172.21.15.106:0/506622986 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:03.215 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f2328a5e700 1 -- 172.21.15.106:0/506622986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318005040 con 0x7f23240f6510 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f2328a5e700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f2318013510 tx=0x7f231800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2318009320 con 0x7f23240f6510 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2318013f00 con 0x7f23240f6510 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2324069fc0 con 0x7f23240f6510 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2318014d90 con 0x7f23240f6510 2024-01-18T07:23:03.216 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.214+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232406a450 con 0x7f23240f6510 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.215+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2318014420 con 0x7f23240f6510 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.215+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324059100 con 0x7f23240f6510 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.215+0000 7f23217fa700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 0x7f2314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.215+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2318040fb0 con 0x7f23240f6510 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.215+0000 7f2323fff700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 0x7f2314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:03.217 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.216+0000 7f2323fff700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 0x7f2314032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:03.220 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.220+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f231801dd50 con 0x7f23240f6510 2024-01-18T07:23:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:00.929451+0000 mgr.smithi084.sihbqg (mgr.14162) 109 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:03.426 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.426+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23240fbf60 con 0x7f23240f6510 2024-01-18T07:23:03.427 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.427+0000 7f23217fa700 1 -- 172.21.15.106:0/506622986 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f231801db70 con 0x7f23240f6510 2024-01-18T07:23:03.427 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:03.427 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 msgr2=0x7f2314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 0x7f2314032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).stop 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 msgr2=0x7f232406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f2318013510 tx=0x7f231800b040).stop 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 shutdown_connections 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f231402ff60 0x7f2314032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 --2- 172.21.15.106:0/506622986 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f23240f6510 0x7f232406b9a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:03.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 >> 172.21.15.106:0/506622986 conn(0x7f23240f1c70 msgr2=0x7f23240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:03.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 shutdown_connections 2024-01-18T07:23:03.430 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:03.429+0000 7f232acc2700 1 -- 172.21.15.106:0/506622986 wait complete. 2024-01-18T07:23:03.430 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:04.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:03.427603+0000 mon.smithi084 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.106:0/506622986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:04.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:04.820 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:02.929797+0000 mgr.smithi084.sihbqg (mgr.14162) 110 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:05.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 -- 172.21.15.106:0/3104127826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 msgr2=0x7fc20c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:05.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 --2- 172.21.15.106:0/3104127826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fc200009fb0 tx=0x7fc200009b30).stop 2024-01-18T07:23:05.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 -- 172.21.15.106:0/3104127826 shutdown_connections 2024-01-18T07:23:05.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 --2- 172.21.15.106:0/3104127826 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:05.942 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 -- 172.21.15.106:0/3104127826 >> 172.21.15.106:0/3104127826 conn(0x7fc20c0f0b70 msgr2=0x7fc20c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 -- 172.21.15.106:0/3104127826 shutdown_connections 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.941+0000 7fc2138ef700 1 -- 172.21.15.106:0/3104127826 wait complete. 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc2138ef700 1 Processor -- start 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc2138ef700 1 -- start start 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc2138ef700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc2138ef700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20c108e30 con 0x7fc20c0f5410 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc21168b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc21168b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:40942/0 (socket says 172.21.15.106:40942) 2024-01-18T07:23:05.943 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc21168b700 1 -- 172.21.15.106:0/4238104905 learned_addr learned my addr 172.21.15.106:0/4238104905 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.942+0000 7fc21168b700 1 -- 172.21.15.106:0/4238104905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc200005040 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc21168b700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc200013510 tx=0x7fc20000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc200009320 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20c109030 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc200013f00 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc200014d90 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.943+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20c1094c0 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2000095f0 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c045ad0 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc1fdffb700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 0x7fc1f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc200040fe0 con 0x7fc20c0f5410 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc210e8a700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 0x7fc1f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:05.945 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.944+0000 7fc210e8a700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 0x7fc1f8032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc2080096a0 tx=0x7fc208006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:05.948 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:05.949+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc20006d2b0 con 0x7fc20c0f5410 2024-01-18T07:23:06.154 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.154+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc20c0fae60 con 0x7fc20c0f5410 2024-01-18T07:23:06.155 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.155+0000 7fc1fdffb700 1 -- 172.21.15.106:0/4238104905 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc200014420 con 0x7fc20c0f5410 2024-01-18T07:23:06.155 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:06.155 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:06.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 msgr2=0x7fc1f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:06.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 0x7fc1f8032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc2080096a0 tx=0x7fc208006b40).stop 2024-01-18T07:23:06.157 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 msgr2=0x7fc20c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fc200013510 tx=0x7fc20000b040).stop 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 shutdown_connections 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc1f802ffb0 0x7fc1f8032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 --2- 172.21.15.106:0/4238104905 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc20c0f5410 0x7fc20c108940 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 >> 172.21.15.106:0/4238104905 conn(0x7fc20c0f0b70 msgr2=0x7fc20c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 shutdown_connections 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:06.157+0000 7fc2138ef700 1 -- 172.21.15.106:0/4238104905 wait complete. 2024-01-18T07:23:06.158 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:04.930171+0000 mgr.smithi084.sihbqg (mgr.14162) 111 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:06.155707+0000 mon.smithi084 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.106:0/4238104905' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:07.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:07.560 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:08.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.740+0000 7fa7f1289700 1 -- 172.21.15.106:0/3769075366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 msgr2=0x7fa7ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:08.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.740+0000 7fa7f1289700 1 --2- 172.21.15.106:0/3769075366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fa7dc009fb0 tx=0x7fa7dc009b30).stop 2024-01-18T07:23:08.742 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.740+0000 7fa7f1289700 1 -- 172.21.15.106:0/3769075366 shutdown_connections 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.740+0000 7fa7f1289700 1 --2- 172.21.15.106:0/3769075366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.740+0000 7fa7f1289700 1 -- 172.21.15.106:0/3769075366 >> 172.21.15.106:0/3769075366 conn(0x7fa7ec0f0b70 msgr2=0x7fa7ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 -- 172.21.15.106:0/3769075366 shutdown_connections 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 -- 172.21.15.106:0/3769075366 wait complete. 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 Processor -- start 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 -- start start 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.741+0000 7fa7f1289700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0fb820 con 0x7fa7ec0f5410 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:40958/0 (socket says 172.21.15.106:40958) 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7eaffd700 1 -- 172.21.15.106:0/79407121 learned_addr learned my addr 172.21.15.106:0/79407121 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7eaffd700 1 -- 172.21.15.106:0/79407121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc005040 con 0x7fa7ec0f5410 2024-01-18T07:23:08.743 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7eaffd700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa7dc013510 tx=0x7fa7dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:08.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7dc009320 con 0x7fa7ec0f5410 2024-01-18T07:23:08.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec0fb9c0 con 0x7fa7ec0f5410 2024-01-18T07:23:08.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7dc013f00 con 0x7fa7ec0f5410 2024-01-18T07:23:08.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7dc014d90 con 0x7fa7ec0f5410 2024-01-18T07:23:08.744 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.742+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec0fbe50 con 0x7fa7ec0f5410 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.743+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7dc0095f0 con 0x7fa7ec0f5410 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.743+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec059100 con 0x7fa7ec0f5410 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.743+0000 7fa7d37fe700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 0x7fa7d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.743+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7dc0410a0 con 0x7fa7ec0f5410 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.744+0000 7fa7ea7fc700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 0x7fa7d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:08.745 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.744+0000 7fa7ea7fc700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 0x7fa7d4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa7e00096a0 tx=0x7fa7e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:08.748 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.748+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7dc01d7f0 con 0x7fa7ec0f5410 2024-01-18T07:23:08.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.951+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7ec0fae60 con 0x7fa7ec0f5410 2024-01-18T07:23:08.952 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.952+0000 7fa7d37fe700 1 -- 172.21.15.106:0/79407121 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7dc010230 con 0x7fa7ec0f5410 2024-01-18T07:23:08.952 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:08.953 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 msgr2=0x7fa7d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 0x7fa7d4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fa7e00096a0 tx=0x7fa7e0006b40).stop 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 msgr2=0x7fa7ec0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fa7dc013510 tx=0x7fa7dc00b040).stop 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 shutdown_connections 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa7d402ff60 0x7fa7d4032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:08.954 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 --2- 172.21.15.106:0/79407121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa7ec0f5410 0x7fa7ec0fd3a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:08.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 >> 172.21.15.106:0/79407121 conn(0x7fa7ec0f0b70 msgr2=0x7fa7ec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:08.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 shutdown_connections 2024-01-18T07:23:08.955 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:08.954+0000 7fa7f1289700 1 -- 172.21.15.106:0/79407121 wait complete. 2024-01-18T07:23:08.955 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:06.930482+0000 mgr.smithi084.sihbqg (mgr.14162) 112 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:10.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:10.365 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:10.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:08.952883+0000 mon.smithi084 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.106:0/79407121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:08.930865+0000 mgr.smithi084.sihbqg (mgr.14162) 113 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 -- 172.21.15.106:0/4188037928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 msgr2=0x7f28d80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 --2- 172.21.15.106:0/4188037928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d80f4be0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f28c8009fb0 tx=0x7f28c8009b30).stop 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 -- 172.21.15.106:0/4188037928 shutdown_connections 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 --2- 172.21.15.106:0/4188037928 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d80f4be0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 -- 172.21.15.106:0/4188037928 >> 172.21.15.106:0/4188037928 conn(0x7f28d80f02e0 msgr2=0x7f28d80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 -- 172.21.15.106:0/4188037928 shutdown_connections 2024-01-18T07:23:11.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.573+0000 7f28dd771700 1 -- 172.21.15.106:0/4188037928 wait complete. 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.574+0000 7f28dd771700 1 Processor -- start 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.574+0000 7f28dd771700 1 -- start start 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.574+0000 7f28dd771700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.574+0000 7f28dd771700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d8106bf0 con 0x7f28d80f47c0 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28d6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28d6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:40968/0 (socket says 172.21.15.106:40968) 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28d6ffd700 1 -- 172.21.15.106:0/1422111165 learned_addr learned my addr 172.21.15.106:0/1422111165 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28d6ffd700 1 -- 172.21.15.106:0/1422111165 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c8005040 con 0x7f28d80f47c0 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28d6ffd700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f28c8013510 tx=0x7f28c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:11.576 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c8009320 con 0x7f28d80f47c0 2024-01-18T07:23:11.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d8106df0 con 0x7f28d80f47c0 2024-01-18T07:23:11.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28c8013f00 con 0x7f28d80f47c0 2024-01-18T07:23:11.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c8014d90 con 0x7f28d80f47c0 2024-01-18T07:23:11.577 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.575+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d8107280 con 0x7f28d80f47c0 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.576+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28c8014420 con 0x7f28d80f47c0 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.576+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d8045ad0 con 0x7f28d80f47c0 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.576+0000 7f28bf7fe700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 0x7f28c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.576+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28c8041000 con 0x7f28d80f47c0 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.577+0000 7f28d67fc700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 0x7f28c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:11.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.577+0000 7f28d67fc700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 0x7f28c0032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f28cc0096a0 tx=0x7f28cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:11.581 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.581+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28c80095f0 con 0x7f28d80f47c0 2024-01-18T07:23:11.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.784+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28d8108130 con 0x7f28d80f47c0 2024-01-18T07:23:11.784 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.784+0000 7f28bf7fe700 1 -- 172.21.15.106:0/1422111165 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f28c801db70 con 0x7f28d80f47c0 2024-01-18T07:23:11.785 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:11.785 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:11.786 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 msgr2=0x7f28c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 0x7f28c0032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f28cc0096a0 tx=0x7f28cc006b40).stop 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 msgr2=0x7f28d8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f28c8013510 tx=0x7f28c800b040).stop 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 shutdown_connections 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28c002ff60 0x7f28c0032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 --2- 172.21.15.106:0/1422111165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28d80f47c0 0x7f28d8106700 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 >> 172.21.15.106:0/1422111165 conn(0x7f28d80f02e0 msgr2=0x7f28d80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 shutdown_connections 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:11.786+0000 7f28dd771700 1 -- 172.21.15.106:0/1422111165 wait complete. 2024-01-18T07:23:11.787 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:11.785003+0000 mon.smithi084 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.106:0/1422111165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:13.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:13.192 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:10.931148+0000 mgr.smithi084.sihbqg (mgr.14162) 114 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 -- 172.21.15.106:0/4036173028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 msgr2=0x7f9e100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 --2- 172.21.15.106:0/4036173028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e100f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f9e08009fb0 tx=0x7f9e08009b30).stop 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 -- 172.21.15.106:0/4036173028 shutdown_connections 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 --2- 172.21.15.106:0/4036173028 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e100f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 -- 172.21.15.106:0/4036173028 >> 172.21.15.106:0/4036173028 conn(0x7f9e100f1c70 msgr2=0x7f9e100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 -- 172.21.15.106:0/4036173028 shutdown_connections 2024-01-18T07:23:14.361 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 -- 172.21.15.106:0/4036173028 wait complete. 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.359+0000 7f9e16180700 1 Processor -- start 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e16180700 1 -- start start 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e16180700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e16180700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10069e20 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e0f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:40980/0 (socket says 172.21.15.106:40980) 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e0f7fe700 1 -- 172.21.15.106:0/4177251532 learned_addr learned my addr 172.21.15.106:0/4177251532 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e0f7fe700 1 -- 172.21.15.106:0/4177251532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08005040 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9e0f7fe700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9e08014000 tx=0x7f9e0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.360+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e080095f0 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e10069fc0 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e08007e70 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e08014ea0 con 0x7f9e100f6510 2024-01-18T07:23:14.362 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1006a450 con 0x7f9e100f6510 2024-01-18T07:23:14.363 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e08025070 con 0x7f9e100f6510 2024-01-18T07:23:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.361+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e10059100 con 0x7f9e100f6510 2024-01-18T07:23:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.362+0000 7f9df7fff700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 0x7f9df8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.362+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e08040a10 con 0x7f9e100f6510 2024-01-18T07:23:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.362+0000 7f9e0effd700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 0x7f9df8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:14.364 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.362+0000 7f9e0effd700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 0x7f9df8032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f9e040096a0 tx=0x7f9e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:14.366 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.366+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e080079c0 con 0x7f9e100f6510 2024-01-18T07:23:14.571 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.571+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e10045ad0 con 0x7f9e100f6510 2024-01-18T07:23:14.572 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.572+0000 7f9df7fff700 1 -- 172.21.15.106:0/4177251532 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e0800f300 con 0x7f9e100f6510 2024-01-18T07:23:14.572 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:14.572 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:14.573 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 msgr2=0x7f9df8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 0x7f9df8032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f9e040096a0 tx=0x7f9e04006b40).stop 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 msgr2=0x7f9e1006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9e08014000 tx=0x7f9e0800b040).stop 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 shutdown_connections 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9df802ff60 0x7f9df8032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 --2- 172.21.15.106:0/4177251532 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9e100f6510 0x7f9e1006b9a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.573+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 >> 172.21.15.106:0/4177251532 conn(0x7f9e100f1c70 msgr2=0x7f9e100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.574+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 shutdown_connections 2024-01-18T07:23:14.574 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:14.574+0000 7f9e16180700 1 -- 172.21.15.106:0/4177251532 wait complete. 2024-01-18T07:23:14.575 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:12.931413+0000 mgr.smithi084.sihbqg (mgr.14162) 115 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:14.572491+0000 mon.smithi084 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.106:0/4177251532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:15.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:15.901 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 -- 172.21.15.106:0/1363919980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 msgr2=0x7fe3f40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/1363919980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f40f7d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fe3f0009fb0 tx=0x7fe3f0009b30).stop 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 -- 172.21.15.106:0/1363919980 shutdown_connections 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/1363919980 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f40f7d00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 -- 172.21.15.106:0/1363919980 >> 172.21.15.106:0/1363919980 conn(0x7fe3f40f0120 msgr2=0x7fe3f40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 -- 172.21.15.106:0/1363919980 shutdown_connections 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 -- 172.21.15.106:0/1363919980 wait complete. 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.048+0000 7fe3fb23a700 1 Processor -- start 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fb23a700 1 -- start start 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fb23a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:17.050 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fb23a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f41069d0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fa238700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fa238700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51990/0 (socket says 172.21.15.106:51990) 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fa238700 1 -- 172.21.15.106:0/816237183 learned_addr learned my addr 172.21.15.106:0/816237183 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fa238700 1 -- 172.21.15.106:0/816237183 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f0005040 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3fa238700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe3f0009830 tx=0x7fe3f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.049+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f0009320 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3f00136c0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f001ee90 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f4106bd0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.051 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f4107060 con 0x7fe3f40f78e0 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f4045ad0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.050+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3f001e420 con 0x7fe3f40f78e0 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.051+0000 7fe3eaffd700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 0x7fe3e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.051+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3f0040fe0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.051+0000 7fe3f9a37700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 0x7fe3e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:17.052 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.051+0000 7fe3f9a37700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 0x7fe3e00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe3e40096a0 tx=0x7fe3e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:17.055 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.055+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3f00095f0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.260+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3f4107f10 con 0x7fe3f40f78e0 2024-01-18T07:23:17.261 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.261+0000 7fe3eaffd700 1 -- 172.21.15.106:0/816237183 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3f002c2d0 con 0x7fe3f40f78e0 2024-01-18T07:23:17.261 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:17.261 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 msgr2=0x7fe3e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 0x7fe3e00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe3e40096a0 tx=0x7fe3e4006b40).stop 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 msgr2=0x7fe3f41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe3f0009830 tx=0x7fe3f000b040).stop 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 shutdown_connections 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3e002ff10 0x7fe3e00323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 --2- 172.21.15.106:0/816237183 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3f40f78e0 0x7fe3f41064e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 >> 172.21.15.106:0/816237183 conn(0x7fe3f40f0120 msgr2=0x7fe3f40f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 shutdown_connections 2024-01-18T07:23:17.263 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:17.263+0000 7fe3fb23a700 1 -- 172.21.15.106:0/816237183 wait complete. 2024-01-18T07:23:17.264 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:14.931666+0000 mgr.smithi084.sihbqg (mgr.14162) 116 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:17.261750+0000 mon.smithi084 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.106:0/816237183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:18.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:18.652 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:16.931985+0000 mgr.smithi084.sihbqg (mgr.14162) 117 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:19.805 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.804+0000 7f41377aa700 1 -- 172.21.15.106:0/3163984739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 msgr2=0x7f41300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.804+0000 7f41377aa700 1 --2- 172.21.15.106:0/3163984739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f41300f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f4120009fb0 tx=0x7f4120009b30).stop 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.804+0000 7f41377aa700 1 -- 172.21.15.106:0/3163984739 shutdown_connections 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.804+0000 7f41377aa700 1 --2- 172.21.15.106:0/3163984739 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f41300f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.804+0000 7f41377aa700 1 -- 172.21.15.106:0/3163984739 >> 172.21.15.106:0/3163984739 conn(0x7f41300f0b70 msgr2=0x7f41300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 -- 172.21.15.106:0/3163984739 shutdown_connections 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 -- 172.21.15.106:0/3163984739 wait complete. 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 Processor -- start 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 -- start start 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:19.806 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.805+0000 7f41377aa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130060220 con 0x7f41300f5410 2024-01-18T07:23:19.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4135546700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:19.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4135546700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:52010/0 (socket says 172.21.15.106:52010) 2024-01-18T07:23:19.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4135546700 1 -- 172.21.15.106:0/749805447 learned_addr learned my addr 172.21.15.106:0/749805447 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:19.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4135546700 1 -- 172.21.15.106:0/749805447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4120005040 con 0x7f41300f5410 2024-01-18T07:23:19.808 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4135546700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f4120013510 tx=0x7f412000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4120009320 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4130060420 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4120013f00 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.806+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4120014d90 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.807+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4130109b00 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41200095f0 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4130059100 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f4125ffb700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 0x7f411c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4120041080 con 0x7f41300f5410 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f4134d45700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 0x7f411c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:19.809 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.808+0000 7f4134d45700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 0x7f411c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f412c0096a0 tx=0x7f412c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:19.812 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:19.812+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f412001d7f0 con 0x7f41300f5410 2024-01-18T07:23:20.017 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.017+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4130045ad0 con 0x7f41300f5410 2024-01-18T07:23:20.018 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.017+0000 7f4125ffb700 1 -- 172.21.15.106:0/749805447 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4120010230 con 0x7f41300f5410 2024-01-18T07:23:20.018 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:20.018 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 msgr2=0x7f411c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 0x7f411c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f412c0096a0 tx=0x7f412c006b40).stop 2024-01-18T07:23:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 msgr2=0x7f4130109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f4120013510 tx=0x7f412000b040).stop 2024-01-18T07:23:20.020 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 shutdown_connections 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f411c02ff60 0x7f411c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 --2- 172.21.15.106:0/749805447 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f41300f5410 0x7f4130109450 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 >> 172.21.15.106:0/749805447 conn(0x7f41300f0b70 msgr2=0x7f41300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 shutdown_connections 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:20.019+0000 7f41377aa700 1 -- 172.21.15.106:0/749805447 wait complete. 2024-01-18T07:23:20.021 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:18.932312+0000 mgr.smithi084.sihbqg (mgr.14162) 118 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:20.018151+0000 mon.smithi084 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.106:0/749805447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:21.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:21.401 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:22.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- 172.21.15.106:0/2975048184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 msgr2=0x7f0a500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:22.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 --2- 172.21.15.106:0/2975048184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a500f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f0a40009fb0 tx=0x7f0a40009b30).stop 2024-01-18T07:23:22.556 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- 172.21.15.106:0/2975048184 shutdown_connections 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 --2- 172.21.15.106:0/2975048184 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a500f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- 172.21.15.106:0/2975048184 >> 172.21.15.106:0/2975048184 conn(0x7f0a500f09a0 msgr2=0x7f0a500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- 172.21.15.106:0/2975048184 shutdown_connections 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- 172.21.15.106:0/2975048184 wait complete. 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 Processor -- start 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- start start 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a57263700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50108bb0 con 0x7f0a500f51a0 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.554+0000 7f0a56261700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a56261700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:52022/0 (socket says 172.21.15.106:52022) 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a56261700 1 -- 172.21.15.106:0/1132547111 learned_addr learned my addr 172.21.15.106:0/1132547111 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a56261700 1 -- 172.21.15.106:0/1132547111 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40005040 con 0x7f0a500f51a0 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a56261700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0a40013510 tx=0x7f0a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a40009320 con 0x7f0a500f51a0 2024-01-18T07:23:22.557 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a50108db0 con 0x7f0a500f51a0 2024-01-18T07:23:22.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a40013f00 con 0x7f0a500f51a0 2024-01-18T07:23:22.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a40014dc0 con 0x7f0a500f51a0 2024-01-18T07:23:22.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.555+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a50109240 con 0x7f0a500f51a0 2024-01-18T07:23:22.558 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.556+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a400095f0 con 0x7f0a500f51a0 2024-01-18T07:23:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.556+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a50045ad0 con 0x7f0a500f51a0 2024-01-18T07:23:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.556+0000 7f0a46ffd700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 0x7f0a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.556+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a40041000 con 0x7f0a500f51a0 2024-01-18T07:23:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.556+0000 7f0a55a60700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 0x7f0a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:22.559 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.557+0000 7f0a55a60700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 0x7f0a3c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0a4c0096a0 tx=0x7f0a4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:22.562 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.561+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a4001c790 con 0x7f0a500f51a0 2024-01-18T07:23:22.767 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.766+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a5010a0f0 con 0x7f0a500f51a0 2024-01-18T07:23:22.768 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.767+0000 7f0a46ffd700 1 -- 172.21.15.106:0/1132547111 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a40010230 con 0x7f0a500f51a0 2024-01-18T07:23:22.768 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:22.768 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 msgr2=0x7f0a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 0x7f0a3c032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0a4c0096a0 tx=0x7f0a4c006b40).stop 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 msgr2=0x7f0a501086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0a40013510 tx=0x7f0a4000b040).stop 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 shutdown_connections 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f0a3c02ff60 0x7f0a3c032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 --2- 172.21.15.106:0/1132547111 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f0a500f51a0 0x7f0a501086c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 >> 172.21.15.106:0/1132547111 conn(0x7f0a500f09a0 msgr2=0x7f0a5005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 shutdown_connections 2024-01-18T07:23:22.770 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:22.769+0000 7f0a57263700 1 -- 172.21.15.106:0/1132547111 wait complete. 2024-01-18T07:23:22.771 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:20.932616+0000 mgr.smithi084.sihbqg (mgr.14162) 119 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:22.768686+0000 mon.smithi084 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.106:0/1132547111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:24.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:24.166 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:25.320 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 -- 172.21.15.106:0/3551514446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 msgr2=0x7fa63c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 --2- 172.21.15.106:0/3551514446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa62c009fb0 tx=0x7fa62c009b30).stop 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 -- 172.21.15.106:0/3551514446 shutdown_connections 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 --2- 172.21.15.106:0/3551514446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 -- 172.21.15.106:0/3551514446 >> 172.21.15.106:0/3551514446 conn(0x7fa63c0f0b70 msgr2=0x7fa63c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 -- 172.21.15.106:0/3551514446 shutdown_connections 2024-01-18T07:23:25.321 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.319+0000 7fa6410bb700 1 -- 172.21.15.106:0/3551514446 wait complete. 2024-01-18T07:23:25.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa6410bb700 1 Processor -- start 2024-01-18T07:23:25.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa6410bb700 1 -- start start 2024-01-18T07:23:25.322 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa6410bb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa6410bb700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c060220 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa63ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa63ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:48352/0 (socket says 172.21.15.106:48352) 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.320+0000 7fa63ad9d700 1 -- 172.21.15.106:0/1538699958 learned_addr learned my addr 172.21.15.106:0/1538699958 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa63ad9d700 1 -- 172.21.15.106:0/1538699958 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c005040 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa63ad9d700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa62c013510 tx=0x7fa62c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa62c009320 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c060420 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa62c013f00 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa62c014d90 con 0x7fa63c0f5410 2024-01-18T07:23:25.323 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.321+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c109b00 con 0x7fa63c0f5410 2024-01-18T07:23:25.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.322+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa62c0095f0 con 0x7fa63c0f5410 2024-01-18T07:23:25.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.322+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c059100 con 0x7fa63c0f5410 2024-01-18T07:23:25.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.322+0000 7fa6237fe700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 0x7fa624032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:25.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.322+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa62c041000 con 0x7fa63c0f5410 2024-01-18T07:23:25.324 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.322+0000 7fa63a59c700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 0x7fa624032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:25.325 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.323+0000 7fa63a59c700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 0x7fa624032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa6300096a0 tx=0x7fa630006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:25.327 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.327+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa62c014420 con 0x7fa63c0f5410 2024-01-18T07:23:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:22.932924+0000 mgr.smithi084.sihbqg (mgr.14162) 120 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:25.534 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.533+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa63c045ad0 con 0x7fa63c0f5410 2024-01-18T07:23:25.535 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.533+0000 7fa6237fe700 1 -- 172.21.15.106:0/1538699958 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa62c010230 con 0x7fa63c0f5410 2024-01-18T07:23:25.535 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:25.535 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 msgr2=0x7fa624032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 0x7fa624032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa6300096a0 tx=0x7fa630006b40).stop 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 msgr2=0x7fa63c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa62c013510 tx=0x7fa62c00b040).stop 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 shutdown_connections 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa62402ff60 0x7fa624032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 --2- 172.21.15.106:0/1538699958 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa63c0f5410 0x7fa63c109450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 >> 172.21.15.106:0/1538699958 conn(0x7fa63c0f0b70 msgr2=0x7fa63c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 shutdown_connections 2024-01-18T07:23:25.537 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:25.535+0000 7fa6410bb700 1 -- 172.21.15.106:0/1538699958 wait complete. 2024-01-18T07:23:25.538 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:26.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:25.535116+0000 mon.smithi084 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.106:0/1538699958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:26.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:26.929 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:24.933248+0000 mgr.smithi084.sihbqg (mgr.14162) 121 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- 172.21.15.106:0/3655181148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 msgr2=0x7f12c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/3655181148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c40f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f12b4009fb0 tx=0x7f12b4009b30).stop 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- 172.21.15.106:0/3655181148 shutdown_connections 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/3655181148 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c40f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- 172.21.15.106:0/3655181148 >> 172.21.15.106:0/3655181148 conn(0x7f12c40f0b70 msgr2=0x7f12c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- 172.21.15.106:0/3655181148 shutdown_connections 2024-01-18T07:23:28.073 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- 172.21.15.106:0/3655181148 wait complete. 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 Processor -- start 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- start start 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.071+0000 7f12c8bc7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c4107240 con 0x7f12c40f5410 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c259c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:48370/0 (socket says 172.21.15.106:48370) 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c259c700 1 -- 172.21.15.106:0/708524206 learned_addr learned my addr 172.21.15.106:0/708524206 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c259c700 1 -- 172.21.15.106:0/708524206 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4005040 con 0x7f12c40f5410 2024-01-18T07:23:28.074 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c259c700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f12b4013510 tx=0x7f12b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:28.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b4009320 con 0x7f12c40f5410 2024-01-18T07:23:28.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c4060220 con 0x7f12c40f5410 2024-01-18T07:23:28.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12b4013f00 con 0x7f12c40f5410 2024-01-18T07:23:28.075 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12b4014d90 con 0x7f12c40f5410 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.072+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c410bd40 con 0x7f12c40f5410 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.073+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12b40263a0 con 0x7f12c40f5410 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.073+0000 7f12b2ffd700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 0x7f12ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.073+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12b40410b0 con 0x7f12c40f5410 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.073+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c4045ad0 con 0x7f12c40f5410 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.073+0000 7f12c1d9b700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 0x7f12ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:28.076 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.074+0000 7f12c1d9b700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 0x7f12ac032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f12b80096a0 tx=0x7f12b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:28.079 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.078+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12b4012920 con 0x7f12c40f5410 2024-01-18T07:23:28.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.283+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12c410c3f0 con 0x7f12c40f5410 2024-01-18T07:23:28.284 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.283+0000 7f12b2ffd700 1 -- 172.21.15.106:0/708524206 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12b402b020 con 0x7f12c40f5410 2024-01-18T07:23:28.285 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:28.285 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 msgr2=0x7f12ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 0x7f12ac032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f12b80096a0 tx=0x7f12b8006b40).stop 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 msgr2=0x7f12c410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f12b4013510 tx=0x7f12b400b040).stop 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 shutdown_connections 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f12ac02ff60 0x7f12ac032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 --2- 172.21.15.106:0/708524206 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f12c40f5410 0x7f12c410b850 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 >> 172.21.15.106:0/708524206 conn(0x7f12c40f0b70 msgr2=0x7f12c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:28.286 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 shutdown_connections 2024-01-18T07:23:28.287 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:28.285+0000 7f12c8bc7700 1 -- 172.21.15.106:0/708524206 wait complete. 2024-01-18T07:23:28.287 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:26.933515+0000 mgr.smithi084.sihbqg (mgr.14162) 122 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:28.284915+0000 mon.smithi084 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.106:0/708524206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:29.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:29.685 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:30.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:28.933792+0000 mgr.smithi084.sihbqg (mgr.14162) 123 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- 172.21.15.106:0/633649401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 msgr2=0x7ff8400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 --2- 172.21.15.106:0/633649401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8400f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7ff834009fb0 tx=0x7ff834009b30).stop 2024-01-18T07:23:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- 172.21.15.106:0/633649401 shutdown_connections 2024-01-18T07:23:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 --2- 172.21.15.106:0/633649401 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8400f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:30.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- 172.21.15.106:0/633649401 >> 172.21.15.106:0/633649401 conn(0x7ff8400f09a0 msgr2=0x7ff8400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- 172.21.15.106:0/633649401 shutdown_connections 2024-01-18T07:23:30.847 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- 172.21.15.106:0/633649401 wait complete. 2024-01-18T07:23:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 Processor -- start 2024-01-18T07:23:30.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- start start 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.845+0000 7ff83f59e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff840060240 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83e59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:48386/0 (socket says 172.21.15.106:48386) 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83e59c700 1 -- 172.21.15.106:0/1099635230 learned_addr learned my addr 172.21.15.106:0/1099635230 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83e59c700 1 -- 172.21.15.106:0/1099635230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff834005040 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83e59c700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff834013510 tx=0x7ff83400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff834009320 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff840060440 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff834013f00 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff834014dc0 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.846+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff840109860 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.847+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8340095f0 con 0x7ff8400f51a0 2024-01-18T07:23:30.849 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.847+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840045ad0 con 0x7ff8400f51a0 2024-01-18T07:23:30.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.847+0000 7ff82effd700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 0x7ff828032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:30.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.847+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff834041000 con 0x7ff8400f51a0 2024-01-18T07:23:30.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.847+0000 7ff83dd9b700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 0x7ff828032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:30.850 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.848+0000 7ff83dd9b700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 0x7ff828032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff8300096a0 tx=0x7ff830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:30.853 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:30.852+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff834014420 con 0x7ff8400f51a0 2024-01-18T07:23:31.060 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.059+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff84010a080 con 0x7ff8400f51a0 2024-01-18T07:23:31.061 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.060+0000 7ff82effd700 1 -- 172.21.15.106:0/1099635230 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff834010230 con 0x7ff8400f51a0 2024-01-18T07:23:31.061 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:31.061 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 msgr2=0x7ff828032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 0x7ff828032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff8300096a0 tx=0x7ff830006b40).stop 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 msgr2=0x7ff8401091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff834013510 tx=0x7ff83400b040).stop 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 shutdown_connections 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff82802ff60 0x7ff828032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 --2- 172.21.15.106:0/1099635230 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8400f51a0 0x7ff8401091b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:31.063 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 >> 172.21.15.106:0/1099635230 conn(0x7ff8400f09a0 msgr2=0x7ff8400f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:31.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 shutdown_connections 2024-01-18T07:23:31.064 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:31.062+0000 7ff83f59e700 1 -- 172.21.15.106:0/1099635230 wait complete. 2024-01-18T07:23:31.064 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:32.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:31.061516+0000 mon.smithi084 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.106:0/1099635230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:32.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:32.472 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:30.934087+0000 mgr.smithi084.sihbqg (mgr.14162) 124 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 -- 172.21.15.106:0/3838696615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 msgr2=0x7ff7cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 --2- 172.21.15.106:0/3838696615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f90f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff7c0009fb0 tx=0x7ff7c0009b30).stop 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 -- 172.21.15.106:0/3838696615 shutdown_connections 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 --2- 172.21.15.106:0/3838696615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f90f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 -- 172.21.15.106:0/3838696615 >> 172.21.15.106:0/3838696615 conn(0x7ff7cc0f0b70 msgr2=0x7ff7cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 -- 172.21.15.106:0/3838696615 shutdown_connections 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 -- 172.21.15.106:0/3838696615 wait complete. 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.631+0000 7ff7d306e700 1 Processor -- start 2024-01-18T07:23:33.634 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d306e700 1 -- start start 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d306e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d306e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0f4ff0 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d0e0a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d0e0a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:48416/0 (socket says 172.21.15.106:48416) 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d0e0a700 1 -- 172.21.15.106:0/1162252171 learned_addr learned my addr 172.21.15.106:0/1162252171 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d0e0a700 1 -- 172.21.15.106:0/1162252171 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0005040 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7d0e0a700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff7c0013510 tx=0x7ff7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.632+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0009320 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c0013f00 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc0f5190 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7c0014dc0 con 0x7ff7cc060280 2024-01-18T07:23:33.635 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc0f5620 con 0x7ff7cc060280 2024-01-18T07:23:33.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7c00095f0 con 0x7ff7cc060280 2024-01-18T07:23:33.636 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.633+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc059100 con 0x7ff7cc060280 2024-01-18T07:23:33.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.634+0000 7ff7c97fa700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 0x7ff7bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:33.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.634+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff7c0040fe0 con 0x7ff7cc060280 2024-01-18T07:23:33.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.634+0000 7ff7cbfff700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 0x7ff7bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:33.637 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.634+0000 7ff7cbfff700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 0x7ff7bc032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff7b80096a0 tx=0x7ff7b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:33.639 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.639+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7c006d2b0 con 0x7ff7cc060280 2024-01-18T07:23:33.845 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.844+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7cc0fbf50 con 0x7ff7cc060280 2024-01-18T07:23:33.846 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.845+0000 7ff7c97fa700 1 -- 172.21.15.106:0/1162252171 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7c0014420 con 0x7ff7cc060280 2024-01-18T07:23:33.846 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:33.846 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 msgr2=0x7ff7bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 0x7ff7bc032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff7b80096a0 tx=0x7ff7b8006b40).stop 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 msgr2=0x7ff7cc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff7c0013510 tx=0x7ff7c000b040).stop 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 shutdown_connections 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff7bc02ffb0 0x7ff7bc032470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 --2- 172.21.15.106:0/1162252171 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff7cc060280 0x7ff7cc0f6b70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 >> 172.21.15.106:0/1162252171 conn(0x7ff7cc0f0b70 msgr2=0x7ff7cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 shutdown_connections 2024-01-18T07:23:33.848 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:33.846+0000 7ff7d306e700 1 -- 172.21.15.106:0/1162252171 wait complete. 2024-01-18T07:23:33.849 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:33.846379+0000 mon.smithi084 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.106:0/1162252171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:35.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:35.279 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:32.934415+0000 mgr.smithi084.sihbqg (mgr.14162) 125 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:36.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:34.929437+0000 mon.smithi084 (mon.0) 209 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:23:36.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:35.034828+0000 mon.smithi084 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:23:36.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:35.042147+0000 mon.smithi084 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:23:36.362 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:35.158282+0000 mon.smithi084 (mon.0) 212 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:36.363 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:35.397974+0000 mon.smithi084 (mon.0) 213 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:36.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 -- 172.21.15.106:0/2891221802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 msgr2=0x7efbf80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:36.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 --2- 172.21.15.106:0/2891221802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf80f4b60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7efbec009fb0 tx=0x7efbec009b30).stop 2024-01-18T07:23:36.546 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 -- 172.21.15.106:0/2891221802 shutdown_connections 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 --2- 172.21.15.106:0/2891221802 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf80f4b60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 -- 172.21.15.106:0/2891221802 >> 172.21.15.106:0/2891221802 conn(0x7efbf80f0260 msgr2=0x7efbf80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 -- 172.21.15.106:0/2891221802 shutdown_connections 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.543+0000 7efbfc96a700 1 -- 172.21.15.106:0/2891221802 wait complete. 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbfc96a700 1 Processor -- start 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbfc96a700 1 -- start start 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbfc96a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbfc96a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf8106bc0 con 0x7efbf80f4740 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbf659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbf659c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51414/0 (socket says 172.21.15.106:51414) 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.544+0000 7efbf659c700 1 -- 172.21.15.106:0/4232043024 learned_addr learned my addr 172.21.15.106:0/4232043024 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbf659c700 1 -- 172.21.15.106:0/4232043024 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbec005040 con 0x7efbf80f4740 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbf659c700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7efbec014000 tx=0x7efbec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:36.547 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbec0095f0 con 0x7efbf80f4740 2024-01-18T07:23:36.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf8106dc0 con 0x7efbf80f4740 2024-01-18T07:23:36.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efbec007e70 con 0x7efbf80f4740 2024-01-18T07:23:36.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efbec014ea0 con 0x7efbf80f4740 2024-01-18T07:23:36.548 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.545+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf8107250 con 0x7efbf80f4740 2024-01-18T07:23:36.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.546+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efbec025070 con 0x7efbf80f4740 2024-01-18T07:23:36.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.546+0000 7efbe6ffd700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 0x7efbe0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:36.549 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.546+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efbec040a40 con 0x7efbf80f4740 2024-01-18T07:23:36.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.546+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf8059100 con 0x7efbf80f4740 2024-01-18T07:23:36.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.546+0000 7efbf5d9b700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 0x7efbe0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:36.550 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.547+0000 7efbf5d9b700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 0x7efbe0032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7efbe80096a0 tx=0x7efbe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:36.551 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.551+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efbec026350 con 0x7efbf80f4740 2024-01-18T07:23:36.776 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.775+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efbf8045ad0 con 0x7efbf80f4740 2024-01-18T07:23:36.777 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.775+0000 7efbe6ffd700 1 -- 172.21.15.106:0/4232043024 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efbec025300 con 0x7efbf80f4740 2024-01-18T07:23:36.777 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:36.777 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:36.778 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 msgr2=0x7efbe0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 0x7efbe0032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7efbe80096a0 tx=0x7efbe8006b40).stop 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 msgr2=0x7efbf81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7efbec014000 tx=0x7efbec00b040).stop 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 shutdown_connections 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7efbe002ff60 0x7efbe0032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:36.779 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 --2- 172.21.15.106:0/4232043024 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7efbf80f4740 0x7efbf81066d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:36.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 >> 172.21.15.106:0/4232043024 conn(0x7efbf80f0260 msgr2=0x7efbf80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:36.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 shutdown_connections 2024-01-18T07:23:36.780 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:36.777+0000 7efbfc96a700 1 -- 172.21.15.106:0/4232043024 wait complete. 2024-01-18T07:23:36.781 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:37.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:34.928838+0000 mgr.smithi084.sihbqg (mgr.14162) 126 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi084 ... 2024-01-18T07:23:37.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:23:37.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:23:37.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:20:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:21:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:22:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:23:37.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:23:37.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:23:37.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi084 ... 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi084 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:20:58Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:21:49Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.138 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:22:27Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:23:37.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:23:37.140 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:23:37.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:23:37.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:34.934625+0000 mgr.smithi084.sihbqg (mgr.14162) 127 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:37.141 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:36.777030+0000 mon.smithi084 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.106:0/4232043024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:38.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:38.234 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:36.934841+0000 mgr.smithi084.sihbqg (mgr.14162) 128 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:36.996798+0000 mon.smithi084 (mon.0) 215 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 -- 172.21.15.106:0/3684591464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 msgr2=0x7fa00c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 --2- 172.21.15.106:0/3684591464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c0f8f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f9ffc009fb0 tx=0x7f9ffc009b30).stop 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 -- 172.21.15.106:0/3684591464 shutdown_connections 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 --2- 172.21.15.106:0/3684591464 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c0f8f80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 -- 172.21.15.106:0/3684591464 >> 172.21.15.106:0/3684591464 conn(0x7fa00c0f09a0 msgr2=0x7fa00c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 -- 172.21.15.106:0/3684591464 shutdown_connections 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 -- 172.21.15.106:0/3684591464 wait complete. 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.403+0000 7fa0116ab700 1 Processor -- start 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa0116ab700 1 -- start start 2024-01-18T07:23:39.406 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa0116ab700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa0116ab700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1069f0 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa00bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa00bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51428/0 (socket says 172.21.15.106:51428) 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa00bfff700 1 -- 172.21.15.106:0/4025529646 learned_addr learned my addr 172.21.15.106:0/4025529646 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa00bfff700 1 -- 172.21.15.106:0/4025529646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc005040 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.404+0000 7fa00bfff700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9ffc013510 tx=0x7f9ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ffc009320 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c106bf0 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ffc013f00 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ffc014d90 con 0x7fa00c060180 2024-01-18T07:23:39.407 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c107080 con 0x7fa00c060180 2024-01-18T07:23:39.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ffc0095f0 con 0x7fa00c060180 2024-01-18T07:23:39.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.405+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c045ad0 con 0x7fa00c060180 2024-01-18T07:23:39.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.406+0000 7fa008ff9700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 0x7f9ff4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:39.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.406+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ffc041050 con 0x7fa00c060180 2024-01-18T07:23:39.408 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.406+0000 7fa00b7fe700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 0x7f9ff4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:39.409 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.406+0000 7fa00b7fe700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 0x7f9ff4032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:39.411 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.410+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ffc014420 con 0x7fa00c060180 2024-01-18T07:23:39.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.619+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa00c107f30 con 0x7fa00c060180 2024-01-18T07:23:39.621 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.620+0000 7fa008ff9700 1 -- 172.21.15.106:0/4025529646 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ffc010230 con 0x7fa00c060180 2024-01-18T07:23:39.622 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:39.622 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:39.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.621+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 msgr2=0x7f9ff4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:39.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.621+0000 7fa0116ab700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 0x7f9ff4032470 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).stop 2024-01-18T07:23:39.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 msgr2=0x7fa00c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:39.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f9ffc013510 tx=0x7f9ffc00b040).stop 2024-01-18T07:23:39.624 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 shutdown_connections 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9ff402ffb0 0x7f9ff4032470 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 --2- 172.21.15.106:0/4025529646 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa00c060180 0x7fa00c106500 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 >> 172.21.15.106:0/4025529646 conn(0x7fa00c0f09a0 msgr2=0x7fa00c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 shutdown_connections 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:39.622+0000 7fa0116ab700 1 -- 172.21.15.106:0/4025529646 wait complete. 2024-01-18T07:23:39.625 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.149680+0000 mon.smithi084 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.435981+0000 mon.smithi084 (mon.0) 217 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.439456+0000 mon.smithi084 (mon.0) 218 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.440732+0000 mon.smithi084 (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi106", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.443573+0000 mon.smithi084 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi106", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:38.444343+0000 mon.smithi084 (mon.0) 221 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:23:39.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:38.444982+0000 mgr.smithi084.sihbqg (mgr.14162) 129 : cephadm [INF] Deploying daemon crash.smithi106 on smithi106 2024-01-18T07:23:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:38.935159+0000 mgr.smithi084.sihbqg (mgr.14162) 130 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:39.621538+0000 mon.smithi084 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.106:0/4025529646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:41.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:41.029 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:42.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:40.716698+0000 mon.smithi084 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:42.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:40.718037+0000 mon.smithi084 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi106.yhaxbc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-18T07:23:42.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:40.720021+0000 mon.smithi084 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi106.yhaxbc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-18T07:23:42.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:40.720772+0000 mon.smithi084 (mon.0) 226 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-18T07:23:42.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:40.721704+0000 mon.smithi084 (mon.0) 227 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:23:42.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:40.722342+0000 mgr.smithi084.sihbqg (mgr.14162) 131 : cephadm [INF] Deploying daemon mgr.smithi106.yhaxbc on smithi106 2024-01-18T07:23:42.391 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 -- 172.21.15.106:0/2459522299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 msgr2=0x7f10800f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 --2- 172.21.15.106:0/2459522299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f10800f5bc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1070009fb0 tx=0x7f1070009b30).stop 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 -- 172.21.15.106:0/2459522299 shutdown_connections 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 --2- 172.21.15.106:0/2459522299 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f10800f5bc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 -- 172.21.15.106:0/2459522299 >> 172.21.15.106:0/2459522299 conn(0x7f10800f00b0 msgr2=0x7f10800f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 -- 172.21.15.106:0/2459522299 shutdown_connections 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.390+0000 7f10855b0700 1 -- 172.21.15.106:0/2459522299 wait complete. 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.391+0000 7f10855b0700 1 Processor -- start 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.391+0000 7f10855b0700 1 -- start start 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.391+0000 7f10855b0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:42.392 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.391+0000 7f10855b0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1080105910 con 0x7f10800f57a0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.392+0000 7f107effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.392+0000 7f107effd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.106:51440/0 (socket says 172.21.15.106:51440) 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.392+0000 7f107effd700 1 -- 172.21.15.106:0/3967871226 learned_addr learned my addr 172.21.15.106:0/3967871226 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.392+0000 7f107effd700 1 -- 172.21.15.106:0/3967871226 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1070005040 con 0x7f10800f57a0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.392+0000 7f107effd700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1070013510 tx=0x7f107000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.393+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1070009320 con 0x7f10800f57a0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.393+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1070013f00 con 0x7f10800f57a0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.393+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1080105b10 con 0x7f10800f57a0 2024-01-18T07:23:42.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.393+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1070014d90 con 0x7f10800f57a0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.393+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1080105fa0 con 0x7f10800f57a0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080045ad0 con 0x7f10800f57a0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f107000f410 con 0x7f10800f57a0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f10777fe700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 0x7f106c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1070043090 con 0x7f10800f57a0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f107e7fc700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 0x7f106c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:42.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.394+0000 7f107e7fc700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 0x7f106c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f10680096a0 tx=0x7f1068006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:42.398 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.398+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1070012d60 con 0x7f10800f57a0 2024-01-18T07:23:42.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.616+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1080106da0 con 0x7f10800f57a0 2024-01-18T07:23:42.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.617+0000 7f10777fe700 1 -- 172.21.15.106:0/3967871226 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1070012980 con 0x7f10800f57a0 2024-01-18T07:23:42.618 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:42.618 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":1,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:18:40.708288Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 msgr2=0x7f106c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 0x7f106c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f10680096a0 tx=0x7f1068006b40).stop 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 msgr2=0x7f1080105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f1070013510 tx=0x7f107000b040).stop 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 shutdown_connections 2024-01-18T07:23:42.619 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f106c02ff60 0x7f106c032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:42.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 --2- 172.21.15.106:0/3967871226 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f10800f57a0 0x7f1080105420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:42.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 >> 172.21.15.106:0/3967871226 conn(0x7f10800f00b0 msgr2=0x7f10800f0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:42.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 shutdown_connections 2024-01-18T07:23:42.620 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:42.619+0000 7f10855b0700 1 -- 172.21.15.106:0/3967871226 wait complete. 2024-01-18T07:23:42.620 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 1 2024-01-18T07:23:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:40.935520+0000 mgr.smithi084.sihbqg (mgr.14162) 132 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:42.486039+0000 mon.smithi084 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:23:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:42.487315+0000 mon.smithi084 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-18T07:23:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:42.488144+0000 mon.smithi084 (mon.0) 230 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:23:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:42.617843+0000 mon.smithi084 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.106:0/3967871226' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:44.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-18T07:23:44.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-18T07:23:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:42.488914+0000 mgr.smithi084.sihbqg (mgr.14162) 133 : cephadm [INF] Deploying daemon mon.smithi106 on smithi106 2024-01-18T07:23:44.751 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:44 smithi106 systemd[1]: Starting Ceph mon.smithi106 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:23:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:42.935841+0000 mgr.smithi084.sihbqg (mgr.14162) 134 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:45.488 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.356+0000 7fa2809e8700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-18T07:23:45.488 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.356+0000 7fa2809e8700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.356+0000 7fa2809e8700 0 pidfile_write: ignore empty --pid-file 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.360+0000 7fa2809e8700 0 load: jerasure load: lrc load: isa 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: RocksDB version: 6.8.1 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Compile date May 13 2021 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: DB SUMMARY 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: CURRENT file: CURRENT 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.489 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi106/store.db dir, Total Num: 0, files: 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi106/store.db: 000003.log size: 511 ; 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.error_if_exists: 0 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.create_if_missing: 0 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.paranoid_checks: 1 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.env: 0x55eef20e71c0 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.fs: Posix File System 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.info_log: 0x55eef40a7960 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-18T07:23:45.490 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.statistics: (nil) 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.use_fsync: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_log_file_size: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.allow_fallocate: 1 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.use_direct_reads: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.db_log_dir: 2024-01-18T07:23:45.491 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi106/store.db 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_subcompactions: 1 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_background_flushes: -1 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.write_buffer_manager: 0x55eef4d1c030 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-18T07:23:45.492 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.unordered_write: 0 2024-01-18T07:23:45.493 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.row_cache: None 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.wal_filter: None 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.preserve_deletes: 0 2024-01-18T07:23:45.495 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.two_write_queues: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.atomic_flush: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.log_readahead_size: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_background_jobs: 2 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_background_compactions: -1 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-18T07:23:45.496 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_open_files: -1 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Compression algorithms supported: 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kZSTD supported: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kXpressCompression supported: 0 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kLZ4Compression supported: 1 2024-01-18T07:23:45.497 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kBZip2Compression supported: 0 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kZlibCompression supported: 1 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: kSnappyCompression supported: 1 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi106/store.db/MANIFEST-000001 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.merge_operator: 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_filter: None 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-18T07:23:45.498 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eef3fbe128) 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cache_index_and_filter_blocks: 1 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: pin_top_level_index_and_filter: 1 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: index_type: 0 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: data_block_index_type: 0 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: index_shortening: 1 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 bash[53052]: 829ed0a3fd939fe36d91366826f50928c6ef328f01ae0ee71175b0f3f908c2c7 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: data_block_hash_table_util_ratio: 0.750000 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: hash_index_allow_collision: 1 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: checksum: 1 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: no_block_cache: 0 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_cache: 0x55eef3ff4f10 2024-01-18T07:23:45.499 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_cache_name: BinnedLRUCache 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_cache_options: 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: capacity : 536870912 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: num_shard_bits : 4 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: strict_capacity_limit : 0 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: high_pri_pool_ratio: 0.000 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_cache_compressed: (nil) 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: persistent_cache: (nil) 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_size: 4096 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_size_deviation: 10 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_restart_interval: 16 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: index_block_restart_interval: 1 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: metadata_block_size: 4096 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: partition_filters: 0 2024-01-18T07:23:45.500 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: use_delta_encoding: 1 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: whole_key_filtering: 1 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: verify_compression: 0 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: read_amp_bytes_per_bit: 0 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: format_version: 2 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: enable_index_compression: 1 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: block_align: 0 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression: NoCompression 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-18T07:23:45.501 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.num_levels: 7 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-18T07:23:45.502 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-18T07:23:45.503 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-18T07:23:45.504 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.table_properties_collectors: 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.inplace_update_support: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.bloom_locality: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.max_successive_merges: 0 2024-01-18T07:23:45.505 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.ttl: 2592000 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.361+0000 7fa2809e8700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.362+0000 7fa2809e8700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi106/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.362+0000 7fa2809e8700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.362+0000 7fa2809e8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562625363030, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.362+0000 7fa2809e8700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.362+0000 7fa2809e8700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.363+0000 7fa2809e8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562625363510, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1437, "table_properties": {"data_size": 523, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705562625, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-18T07:23:45.506 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.363+0000 7fa2809e8700 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.363+0000 7fa2809e8700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705562625363887, "job": 1, "event": "recovery_finished"} 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.364+0000 7fa2809e8700 4 rocksdb: DB pointer 0x55eef40b9800 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.364+0000 7fa267800700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.364+0000 7fa267800700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** DB Stats ** 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-18T07:23:45.507 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** Compaction Stats [default] ** 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** Compaction Stats [default] ** 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:23:45.508 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** Compaction Stats [default] ** 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: L0 1/0 1.40 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Sum 1/0 1.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.509 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** Compaction Stats [default] ** 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(Total Files): cumulative 0, interval 0 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: AddFile(Keys): cumulative 0, interval 0 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-18T07:23:45.510 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ** File Read Latency Histogram By Level [default] ** 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.364+0000 7fa2809e8700 0 mon.smithi106 does not exist in monmap, will attempt to join an existing cluster 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.364+0000 7fa2809e8700 0 using public_addr v2:172.21.15.106:0/0 -> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.365+0000 7fa2809e8700 0 starting mon.smithi106 rank -1 at public addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] at bind addrs [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi106 fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.366+0000 7fa2809e8700 1 mon.smithi106@-1(???) e0 preinit fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 systemd[1]: Started Ceph mon.smithi106 for bf9448d2-b5d1-11ee-95ae-87774f69a715. 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).mds e1 new map 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).mds e1 print_map 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: e1 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: legacy client fscid: -1 2024-01-18T07:23:45.511 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: No filesystems configured 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.414+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.416+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.416+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.416+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.416+0000 7fa26a005700 0 mon.smithi106@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-01-18T07:23:45.512 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:42.935841+0000 mgr.smithi084.sihbqg (mgr.14162) 134 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.417+0000 7fa26a005700 1 mon.smithi106@-1(synchronizing).paxosservice(auth 1..7) refresh upgraded, format 0 -> 3 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expand map: {default=false} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta from 'false' to 'false' 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expanded map: {default=false} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expand map: {default=info} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta from 'info' to 'info' 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expanded map: {default=info} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expand map: {default=daemon} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expand map: {default=debug} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 20 expand_channel_meta expanded map: {default=debug} 2024-01-18T07:23:45.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:23:45.418+0000 7fa26a005700 10 mon.smithi106@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-18T07:23:50.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:44.936151+0000 mgr.smithi084.sihbqg (mgr.14162) 135 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:45.423209+0000 mgr.smithi084.sihbqg (mgr.14162) 136 : cephadm [INF] Deploying daemon node-exporter.smithi106 on smithi106 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:45.424600+0000 mon.smithi084 (mon.0) 234 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:45.424804+0000 mon.smithi084 (mon.0) 235 : cluster [INF] mon.smithi084 calling monitor election 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:45.437770+0000 mon.smithi084 (mon.0) 236 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:46.421068+0000 mon.smithi084 (mon.0) 237 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:46.936526+0000 mgr.smithi084.sihbqg (mgr.14162) 137 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:47.421018+0000 mon.smithi084 (mon.0) 238 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:47.422380+0000 mon.smithi106 (mon.1) 1 : cluster [INF] mon.smithi106 calling monitor election 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:47.530372+0000 mon.smithi084 (mon.0) 239 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi106.yhaxbc/crt"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:48.421263+0000 mon.smithi084 (mon.0) 240 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:48.936835+0000 mgr.smithi084.sihbqg (mgr.14162) 138 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:49.421492+0000 mon.smithi084 (mon.0) 241 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:50.421543+0000 mon.smithi084 (mon.0) 242 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.451867+0000 mon.smithi084 (mon.0) 243 : cluster [INF] mon.smithi084 is new leader, mons smithi084,smithi106 in quorum (ranks 0,1) 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.457957+0000 mon.smithi084 (mon.0) 244 : cluster [DBG] monmap e2: 2 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0],smithi106=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.458029+0000 mon.smithi084 (mon.0) 245 : cluster [DBG] fsmap 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.458090+0000 mon.smithi084 (mon.0) 246 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.458332+0000 mon.smithi084 (mon.0) 247 : cluster [DBG] mgrmap e17: smithi084.sihbqg(active, since 4m) 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.459343+0000 mon.smithi084 (mon.0) 248 : cluster [DBG] Standby manager daemon smithi106.yhaxbc started 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:50.460675+0000 mon.smithi084 (mon.0) 249 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.462920+0000 mon.smithi084 (mon.0) 250 : cluster [INF] overall HEALTH_OK 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:50.463030+0000 mon.smithi084 (mon.0) 251 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi106.yhaxbc/key"}]: dispatch 2024-01-18T07:23:50.884 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:50.463507+0000 mon.smithi084 (mon.0) 252 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-18T07:23:50.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:44.936151+0000 mgr.smithi084.sihbqg (mgr.14162) 135 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:23:45.423209+0000 mgr.smithi084.sihbqg (mgr.14162) 136 : cephadm [INF] Deploying daemon node-exporter.smithi106 on smithi106 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:45.424600+0000 mon.smithi084 (mon.0) 234 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi084"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:45.424804+0000 mon.smithi084 (mon.0) 235 : cluster [INF] mon.smithi084 calling monitor election 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:45.437770+0000 mon.smithi084 (mon.0) 236 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:46.421068+0000 mon.smithi084 (mon.0) 237 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:46.936526+0000 mgr.smithi084.sihbqg (mgr.14162) 137 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:47.421018+0000 mon.smithi084 (mon.0) 238 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:47.422380+0000 mon.smithi106 (mon.1) 1 : cluster [INF] mon.smithi106 calling monitor election 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:47.530372+0000 mon.smithi084 (mon.0) 239 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi106.yhaxbc/crt"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:48.421263+0000 mon.smithi084 (mon.0) 240 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:48.936835+0000 mgr.smithi084.sihbqg (mgr.14162) 138 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:49.421492+0000 mon.smithi084 (mon.0) 241 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:50.421543+0000 mon.smithi084 (mon.0) 242 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.451867+0000 mon.smithi084 (mon.0) 243 : cluster [INF] mon.smithi084 is new leader, mons smithi084,smithi106 in quorum (ranks 0,1) 2024-01-18T07:23:50.916 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.457957+0000 mon.smithi084 (mon.0) 244 : cluster [DBG] monmap e2: 2 mons at {smithi084=[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0],smithi106=[v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0]} 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.458029+0000 mon.smithi084 (mon.0) 245 : cluster [DBG] fsmap 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.458090+0000 mon.smithi084 (mon.0) 246 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.458332+0000 mon.smithi084 (mon.0) 247 : cluster [DBG] mgrmap e17: smithi084.sihbqg(active, since 4m) 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.459343+0000 mon.smithi084 (mon.0) 248 : cluster [DBG] Standby manager daemon smithi106.yhaxbc started 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:50.460675+0000 mon.smithi084 (mon.0) 249 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.462920+0000 mon.smithi084 (mon.0) 250 : cluster [INF] overall HEALTH_OK 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:50.463030+0000 mon.smithi084 (mon.0) 251 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi106.yhaxbc/key"}]: dispatch 2024-01-18T07:23:50.917 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:50.463507+0000 mon.smithi084 (mon.0) 252 : audit [DBG] from='mgr.? 172.21.15.106:0/637395097' entity='mgr.smithi106.yhaxbc' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 -- 172.21.15.106:0/1764185651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f70005e80 msgr2=0x7f9f700082f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/1764185651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f70005e80 0x7f9f700082f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f9f880f6170 tx=0x7f9f78002040).stop 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 -- 172.21.15.106:0/1764185651 shutdown_connections 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/1764185651 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f70005e80 0x7f9f700082f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 -- 172.21.15.106:0/1764185651 >> 172.21.15.106:0/1764185651 conn(0x7f9f880f0b70 msgr2=0x7f9f880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 -- 172.21.15.106:0/1764185651 shutdown_connections 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.667+0000 7f9f8e40b700 1 -- 172.21.15.106:0/1764185651 wait complete. 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.669+0000 7f9f8e40b700 1 Processor -- start 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f8e40b700 1 -- start start 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f8e40b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:51.671 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f8e40b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 0x7f9f88107b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f8e40b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881071a0 con 0x7f9f88106c40 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f8e40b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881072e0 con 0x7f9f70005e80 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:41856/0 (socket says 172.21.15.106:41856) 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 -- 172.21.15.106:0/2080682911 learned_addr learned my addr 172.21.15.106:0/2080682911 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f877fe700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 0x7f9f88107b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 -- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 msgr2=0x7f9f88106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 -- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 msgr2=0x7f9f88106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f87fff700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f877fe700 1 -- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 msgr2=0x7f9f88106750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.670+0000 7f9f877fe700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f877fe700 1 -- 172.21.15.106:0/2080682911 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f78005040 con 0x7f9f88106c40 2024-01-18T07:23:51.672 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f877fe700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 0x7f9f88107b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9f7c009390 tx=0x7f9f7c006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c0115e0 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f7c011bf0 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88108140 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f7c010bf0 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.671+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f88108700 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.672+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f88059100 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.672+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9f7c018030 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.673+0000 7f9f84ff9700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 0x7f9f7405e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.673+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f7c074ba0 con 0x7f9f88106c40 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.673+0000 7f9f87fff700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 0x7f9f7405e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:51.674 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.674+0000 7f9f87fff700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 0x7f9f7405e8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f9f78004fd0 tx=0x7f9f78002040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:51.677 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.678+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f7c0452b0 con 0x7f9f88106c40 2024-01-18T07:23:51.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.518194+0000 mon.smithi084 (mon.0) 253 : cluster [DBG] mgrmap e18: smithi084.sihbqg(active, since 4m), standbys: smithi106.yhaxbc 2024-01-18T07:23:51.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:50.518335+0000 mon.smithi084 (mon.0) 254 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi106.yhaxbc", "id": "smithi106.yhaxbc"}]: dispatch 2024-01-18T07:23:51.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:51.421897+0000 mon.smithi084 (mon.0) 255 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:51.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.888+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f880f88c0 con 0x7f9f88106c40 2024-01-18T07:23:51.888 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.888+0000 7f9f84ff9700 1 -- 172.21.15.106:0/2080682911 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+968 (secure 0 0 0) 0x7f9f7c016020 con 0x7f9f88106c40 2024-01-18T07:23:51.889 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-01-18T07:23:51.889 INFO:teuthology.orchestra.run.smithi106.stdout:{"epoch":2,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","modified":"2024-01-18T07:23:45.422603Z","created":"2024-01-18T07:18:40.708288Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi084","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi106","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-18T07:23:51.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 msgr2=0x7f9f7405e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:51.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 0x7f9f7405e8b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f9f78004fd0 tx=0x7f9f78002040).stop 2024-01-18T07:23:51.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 msgr2=0x7f9f88107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:51.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 0x7f9f88107b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f9f7c009390 tx=0x7f9f7c006e20).stop 2024-01-18T07:23:51.890 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 shutdown_connections 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9f7405c3f0 0x7f9f7405e8b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9f70005e80 0x7f9f88106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 --2- 172.21.15.106:0/2080682911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9f88106c40 0x7f9f88107b30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 >> 172.21.15.106:0/2080682911 conn(0x7f9f880f0b70 msgr2=0x7f9f8805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 shutdown_connections 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:23:51.890+0000 7f9f8e40b700 1 -- 172.21.15.106:0/2080682911 wait complete. 2024-01-18T07:23:51.891 INFO:teuthology.orchestra.run.smithi106.stderr:dumped monmap epoch 2 2024-01-18T07:23:51.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.518194+0000 mon.smithi084 (mon.0) 253 : cluster [DBG] mgrmap e18: smithi084.sihbqg(active, since 4m), standbys: smithi106.yhaxbc 2024-01-18T07:23:51.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:50.518335+0000 mon.smithi084 (mon.0) 254 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr metadata", "who": "smithi106.yhaxbc", "id": "smithi106.yhaxbc"}]: dispatch 2024-01-18T07:23:51.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:51.421897+0000 mon.smithi084 (mon.0) 255 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mon metadata", "id": "smithi106"}]: dispatch 2024-01-18T07:23:52.279 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-18T07:23:52.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph config generate-minimal-conf 2024-01-18T07:23:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:50.937149+0000 mgr.smithi084.sihbqg (mgr.14162) 139 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:51.888976+0000 mon.smithi084 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.106:0/2080682911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:52.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:50.937149+0000 mgr.smithi084.sihbqg (mgr.14162) 139 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:52.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:51.888976+0000 mon.smithi084 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.106:0/2080682911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- 172.21.15.84:0/3178163236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 msgr2=0x7ffb040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 --2- 172.21.15.84:0/3178163236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb040f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ffaf4009fb0 tx=0x7ffaf4009b30).stop 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- 172.21.15.84:0/3178163236 shutdown_connections 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 --2- 172.21.15.84:0/3178163236 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb040f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- 172.21.15.84:0/3178163236 >> 172.21.15.84:0/3178163236 conn(0x7ffb040f1c70 msgr2=0x7ffb040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:53.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- 172.21.15.84:0/3178163236 shutdown_connections 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- 172.21.15.84:0/3178163236 wait complete. 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 Processor -- start 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.511+0000 7ffb0b979700 1 -- start start 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb04106ca0 0x7ffb04107b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04107200 con 0x7ffb040f6510 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb04107340 con 0x7ffb04106ca0 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34034/0 (socket says 172.21.15.84:34034) 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 -- 172.21.15.84:0/2063662260 learned_addr learned my addr 172.21.15.84:0/2063662260 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 -- 172.21.15.84:0/2063662260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb04106ca0 msgr2=0x7ffb04107b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb04106ca0 0x7ffb04107b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 -- 172.21.15.84:0/2063662260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf4005040 con 0x7ffb040f6510 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb09715700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ffaf4002040 tx=0x7ffaf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:53.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaf4019070 con 0x7ffb040f6510 2024-01-18T07:23:53.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb04108140 con 0x7ffb040f6510 2024-01-18T07:23:53.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffaf4007c70 con 0x7ffb040f6510 2024-01-18T07:23:53.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaf401d620 con 0x7ffb040f6510 2024-01-18T07:23:53.516 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.512+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb041086a0 con 0x7ffb040f6510 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.513+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb04045ad0 con 0x7ffb040f6510 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.514+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ffaf4025070 con 0x7ffb040f6510 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.514+0000 7ffaf9ffb700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 0x7ffaf005e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.514+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffaf40762c0 con 0x7ffb040f6510 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.514+0000 7ffb08f14700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 0x7ffaf005e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:53.517 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.514+0000 7ffb08f14700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 0x7ffaf005e870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ffb000095b0 tx=0x7ffb00006ae0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:53.519 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.518+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffaf4049f90 con 0x7ffb040f6510 2024-01-18T07:23:53.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.670+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ffb040008d0 con 0x7ffb040f6510 2024-01-18T07:23:53.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.670+0000 7ffaf9ffb700 1 -- 172.21.15.84:0/2063662260 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v12) v1 ==== 76+0+225 (secure 0 0 0) 0x7ffaf404d5b0 con 0x7ffb040f6510 2024-01-18T07:23:53.672 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:23:53.672 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-01-18T07:23:53.672 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = bf9448d2-b5d1-11ee-95ae-87774f69a715 2024-01-18T07:23:53.672 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] 2024-01-18T07:23:53.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 msgr2=0x7ffaf005e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:53.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 0x7ffaf005e870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ffb000095b0 tx=0x7ffb00006ae0).stop 2024-01-18T07:23:53.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 msgr2=0x7ffb041067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:53.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ffaf4002040 tx=0x7ffaf400b040).stop 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 shutdown_connections 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ffb040f6510 0x7ffb041067b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ffaf005c3b0 0x7ffaf005e870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 --2- 172.21.15.84:0/2063662260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ffb04106ca0 0x7ffb04107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 >> 172.21.15.84:0/2063662260 conn(0x7ffb040f1c70 msgr2=0x7ffb040f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 shutdown_connections 2024-01-18T07:23:53.674 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:53.672+0000 7ffb0b979700 1 -- 172.21.15.84:0/2063662260 wait complete. 2024-01-18T07:23:54.062 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-18T07:23:54.063 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:23:54.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-01-18T07:23:54.092 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:23:54.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-18T07:23:54.159 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:23:54.159 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-01-18T07:23:54.197 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:23:54.198 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-18T07:23:54.270 INFO:tasks.cephadm:Deploying OSDs... 2024-01-18T07:23:54.270 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:23:54.271 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-01-18T07:23:54.288 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-18T07:23:54.289 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 161191 Links: 1 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-18 07:20:15.287753129 +0000 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-18 07:14:32.093727564 +0000 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-18 07:14:32.093727564 +0000 2024-01-18T07:23:54.346 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-18T07:23:54.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-18T07:23:54.412 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-18T07:23:54.412 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-18T07:23:54.412 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000238481 s, 2.1 MB/s 2024-01-18T07:23:54.413 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-18T07:23:54.472 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 161249 Links: 1 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-18 07:20:15.287753129 +0000 2024-01-18T07:23:54.530 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-18 07:14:32.421721755 +0000 2024-01-18T07:23:54.531 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-18 07:14:32.421721755 +0000 2024-01-18T07:23:54.531 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-18T07:23:54.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-18T07:23:54.596 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-18T07:23:54.596 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-18T07:23:54.596 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000213097 s, 2.4 MB/s 2024-01-18T07:23:54.597 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-18T07:23:54.656 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-01-18T07:23:54.714 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-18T07:23:54.715 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:54.715 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 161297 Links: 1 2024-01-18T07:23:54.715 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:54.715 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:54.715 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-18 07:20:15.287753129 +0000 2024-01-18T07:23:54.716 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-18 07:14:32.761715733 +0000 2024-01-18T07:23:54.716 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-18 07:14:32.761715733 +0000 2024-01-18T07:23:54.716 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-18T07:23:54.716 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-18T07:23:54.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:52.937383+0000 mgr.smithi084.sihbqg (mgr.14162) 140 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:54.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:53.672378+0000 mon.smithi084 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.84:0/2063662260' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:23:54.780 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-18T07:23:54.780 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-18T07:23:54.780 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000167258 s, 3.1 MB/s 2024-01-18T07:23:54.781 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-18T07:23:54.840 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-01-18T07:23:54.897 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d Inode: 166203 Links: 1 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-01-18 07:20:15.287753129 +0000 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-01-18 07:14:33.098709765 +0000 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-01-18 07:14:33.098709765 +0000 2024-01-18T07:23:54.898 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: - 2024-01-18T07:23:54.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-18T07:23:54.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:52.937383+0000 mgr.smithi084.sihbqg (mgr.14162) 140 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:54.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:53.672378+0000 mon.smithi084 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.84:0/2063662260' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:23:54.963 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-01-18T07:23:54.964 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-01-18T07:23:54.964 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000246165 s, 2.1 MB/s 2024-01-18T07:23:54.965 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-18T07:23:55.024 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:23:55.024 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-01-18T07:23:55.047 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-18T07:23:55.047 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 161307 Links: 1 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-18 07:23:45.871383789 +0000 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-18 07:14:32.070878619 +0000 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-18 07:14:32.070878619 +0000 2024-01-18T07:23:55.109 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-18T07:23:55.110 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-18T07:23:55.181 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-18T07:23:55.181 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-18T07:23:55.181 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000284544 s, 1.8 MB/s 2024-01-18T07:23:55.182 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-18T07:23:55.245 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 165930 Links: 1 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-18 07:23:45.871383789 +0000 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-18 07:14:32.392870879 +0000 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-18 07:14:32.392870879 +0000 2024-01-18T07:23:55.307 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-18T07:23:55.308 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-18T07:23:55.379 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-18T07:23:55.379 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-18T07:23:55.379 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000235684 s, 2.2 MB/s 2024-01-18T07:23:55.380 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-18T07:23:55.443 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 165043 Links: 1 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-18 07:23:45.871383789 +0000 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-18 07:14:32.730862754 +0000 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-18 07:14:32.730862754 +0000 2024-01-18T07:23:55.505 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-18T07:23:55.506 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-18T07:23:55.577 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-18T07:23:55.577 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-18T07:23:55.577 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000201657 s, 2.5 MB/s 2024-01-18T07:23:55.578 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-18T07:23:55.642 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 6h/6d Inode: 165098 Links: 1 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-01-18 07:23:45.872383765 +0000 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-01-18 07:14:33.078854390 +0000 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-01-18 07:14:33.078854390 +0000 2024-01-18T07:23:55.704 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: - 2024-01-18T07:23:55.704 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-18T07:23:55.775 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-01-18T07:23:55.775 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-01-18T07:23:55.776 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000270033 s, 1.9 MB/s 2024-01-18T07:23:55.777 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-18T07:23:55.840 INFO:tasks.cephadm:Deploying osd.0 on smithi084 with /dev/vg_nvme/lv_4... 2024-01-18T07:23:55.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-18T07:23:57.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:54.937607+0000 mgr.smithi084.sihbqg (mgr.14162) 141 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:57.376 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-18T07:23:57.376 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-18T07:23:57.376 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-01-18T07:23:57.376 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-01-18T07:23:57.376 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128543 s, 816 MB/s 2024-01-18T07:23:57.377 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-01-18T07:23:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:54.937607+0000 mgr.smithi084.sihbqg (mgr.14162) 141 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:57.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi084:vg_nvme/lv_4 2024-01-18T07:23:58.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 -- 172.21.15.84:0/1874270770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f7980 msgr2=0x7fe4040f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:58.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 --2- 172.21.15.84:0/1874270770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f7980 0x7fe4040f7de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fe3f4004660 tx=0x7fe3f4009b30).stop 2024-01-18T07:23:58.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 -- 172.21.15.84:0/1874270770 shutdown_connections 2024-01-18T07:23:58.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 --2- 172.21.15.84:0/1874270770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f7980 0x7fe4040f7de0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:58.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 --2- 172.21.15.84:0/1874270770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f66e0 0x7fe4040f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:58.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.878+0000 7fe40b838700 1 -- 172.21.15.84:0/1874270770 >> 172.21.15.84:0/1874270770 conn(0x7fe4040f1e40 msgr2=0x7fe4040f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:23:58.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 -- 172.21.15.84:0/1874270770 shutdown_connections 2024-01-18T07:23:58.881 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 -- 172.21.15.84:0/1874270770 wait complete. 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 Processor -- start 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 -- start start 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f7980 0x7fe404107470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404107a20 con 0x7fe4040f66e0 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe40b838700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404107b60 con 0x7fe4040f7980 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe4095d4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.879+0000 7fe4095d4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35456/0 (socket says 172.21.15.84:35456) 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe4095d4700 1 -- 172.21.15.84:0/1578238301 learned_addr learned my addr 172.21.15.84:0/1578238301 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:23:58.883 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe408dd3700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f7980 0x7fe404107470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe4095d4700 1 -- 172.21.15.84:0/1578238301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f7980 msgr2=0x7fe404107470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe4095d4700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f7980 0x7fe404107470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe4095d4700 1 -- 172.21.15.84:0/1578238301 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4005040 con 0x7fe4040f66e0 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe4095d4700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe40000c530 tx=0x7fe400007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe40000f650 con 0x7fe4040f66e0 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe40000fc60 con 0x7fe4040f66e0 2024-01-18T07:23:58.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe404108500 con 0x7fe4040f66e0 2024-01-18T07:23:58.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe400012d20 con 0x7fe4040f66e0 2024-01-18T07:23:58.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.880+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe404108ac0 con 0x7fe4040f66e0 2024-01-18T07:23:58.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.881+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe404045ad0 con 0x7fe4040f66e0 2024-01-18T07:23:58.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.881+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe400019030 con 0x7fe4040f66e0 2024-01-18T07:23:58.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.882+0000 7fe3f9ffb700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 0x7fe3f005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:23:58.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.882+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe400074b70 con 0x7fe4040f66e0 2024-01-18T07:23:58.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.882+0000 7fe408dd3700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 0x7fe3f005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:23:58.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.882+0000 7fe408dd3700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 0x7fe3f005e770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe3f4013580 tx=0x7fe3f4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:23:58.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:58.886+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4000462b0 con 0x7fe4040f66e0 2024-01-18T07:23:59.062 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:23:59.060+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe404108d40 con 0x7fe3f005c2b0 2024-01-18T07:23:59.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:56.937945+0000 mgr.smithi084.sihbqg (mgr.14162) 142 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:23:59.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:56.937945+0000 mgr.smithi084.sihbqg (mgr.14162) 142 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:00.049 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:59.064227+0000 mon.smithi084 (mon.0) 258 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:00.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:59.065624+0000 mon.smithi084 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:00.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:23:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:59.066098+0000 mon.smithi084 (mon.0) 260 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:00.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:59.064227+0000 mon.smithi084 (mon.0) 258 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:00.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:59.065624+0000 mon.smithi084 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:00.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:23:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:59.066098+0000 mon.smithi084 (mon.0) 260 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:01.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:23:58.938181+0000 mgr.smithi084.sihbqg (mgr.14162) 143 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:23:59.063384+0000 mgr.smithi084.sihbqg (mgr.14162) 144 : audit [DBG] from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:23:59.065416+0000 mgr.smithi084.sihbqg (mgr.14162) 145 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:00.761633+0000 mon.smithi084 (mon.0) 261 : audit [INF] from='client.? 172.21.15.84:0/2798574064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2490ef18-5987-444d-80e1-a7ee0dd798a3"}]: dispatch 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:00.765363+0000 mon.smithi084 (mon.0) 262 : audit [INF] from='client.? 172.21.15.84:0/2798574064' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2490ef18-5987-444d-80e1-a7ee0dd798a3"}]': finished 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:00.765417+0000 mon.smithi084 (mon.0) 263 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-18T07:24:01.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:00.765516+0000 mon.smithi084 (mon.0) 264 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:23:58.938181+0000 mgr.smithi084.sihbqg (mgr.14162) 143 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:23:59.063384+0000 mgr.smithi084.sihbqg (mgr.14162) 144 : audit [DBG] from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:23:59.065416+0000 mgr.smithi084.sihbqg (mgr.14162) 145 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:00.761633+0000 mon.smithi084 (mon.0) 261 : audit [INF] from='client.? 172.21.15.84:0/2798574064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2490ef18-5987-444d-80e1-a7ee0dd798a3"}]: dispatch 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:00.765363+0000 mon.smithi084 (mon.0) 262 : audit [INF] from='client.? 172.21.15.84:0/2798574064' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2490ef18-5987-444d-80e1-a7ee0dd798a3"}]': finished 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:00.765417+0000 mon.smithi084 (mon.0) 263 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-01-18T07:24:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:00.765516+0000 mon.smithi084 (mon.0) 264 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:01.174850+0000 mon.smithi084 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.84:0/2191795228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:01.174850+0000 mon.smithi084 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.84:0/2191795228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:00.938407+0000 mgr.smithi084.sihbqg (mgr.14162) 146 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:03.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:00.938407+0000 mgr.smithi084.sihbqg (mgr.14162) 146 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:05.097 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:02.938742+0000 mgr.smithi084.sihbqg (mgr.14162) 147 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:02.938742+0000 mgr.smithi084.sihbqg (mgr.14162) 147 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:07.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:04.939040+0000 mgr.smithi084.sihbqg (mgr.14162) 148 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:07.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:06.464370+0000 mon.smithi084 (mon.0) 266 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-18T07:24:07.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:06.464914+0000 mon.smithi084 (mon.0) 267 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:07.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:04.939040+0000 mgr.smithi084.sihbqg (mgr.14162) 148 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:07.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:06.464370+0000 mon.smithi084 (mon.0) 266 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-18T07:24:07.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:06.464914+0000 mon.smithi084 (mon.0) 267 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:08.103 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:06.465325+0000 mgr.smithi084.sihbqg (mgr.14162) 149 : cephadm [INF] Deploying daemon osd.0 on smithi084 2024-01-18T07:24:08.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:06.465325+0000 mgr.smithi084.sihbqg (mgr.14162) 149 : cephadm [INF] Deploying daemon osd.0 on smithi084 2024-01-18T07:24:08.795 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 0 on host 'smithi084' 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.793+0000 7fe3f9ffb700 1 -- 172.21.15.84:0/1578238301 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe404108d40 con 0x7fe3f005c2b0 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 msgr2=0x7fe3f005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 0x7fe3f005e770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fe3f4013580 tx=0x7fe3f4009b30).stop 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 msgr2=0x7fe404106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fe40000c530 tx=0x7fe400007e10).stop 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 shutdown_connections 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4040f66e0 0x7fe404106f80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3f005c2b0 0x7fe3f005e770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 --2- 172.21.15.84:0/1578238301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4040f7980 0x7fe404107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:08.796 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 >> 172.21.15.84:0/1578238301 conn(0x7fe4040f1e40 msgr2=0x7fe4040fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:08.797 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 shutdown_connections 2024-01-18T07:24:08.797 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:08.795+0000 7fe40b838700 1 -- 172.21.15.84:0/1578238301 wait complete. 2024-01-18T07:24:09.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:06.939362+0000 mgr.smithi084.sihbqg (mgr.14162) 150 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:09.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:08.789593+0000 mon.smithi084 (mon.0) 268 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:09.128 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:08.793078+0000 mon.smithi084 (mon.0) 269 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:09.149 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.0.service 2024-01-18T07:24:09.151 INFO:tasks.cephadm:Deploying osd.1 on smithi084 with /dev/vg_nvme/lv_3... 2024-01-18T07:24:09.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-18T07:24:09.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:06.939362+0000 mgr.smithi084.sihbqg (mgr.14162) 150 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:09.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:08.789593+0000 mon.smithi084 (mon.0) 268 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:09.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:08.793078+0000 mon.smithi084 (mon.0) 269 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:09.180 INFO:journalctl@ceph.osd.0.smithi084.stdout:-- Logs begin at Thu 2024-01-18 07:09:07 UTC. -- 2024-01-18T07:24:09.382 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:24:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:24:09.255+0000 7f96529d3080 -1 Falling back to public interface 2024-01-18T07:24:10.706 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-18T07:24:10.706 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-18T07:24:10.706 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-01-18T07:24:10.706 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-01-18T07:24:10.706 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122589 s, 855 MB/s 2024-01-18T07:24:10.707 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-01-18T07:24:11.067 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:08.939834+0000 mgr.smithi084.sihbqg (mgr.14162) 151 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:11.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi084:vg_nvme/lv_3 2024-01-18T07:24:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:08.939834+0000 mgr.smithi084.sihbqg (mgr.14162) 151 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:12.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 -- 172.21.15.84:0/2154866623 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f5430 msgr2=0x7f61e00f5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:12.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 --2- 172.21.15.84:0/2154866623 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f5430 0x7f61e00f5850 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f61d0004660 tx=0x7f61d0009b30).stop 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 -- 172.21.15.84:0/2154866623 shutdown_connections 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 --2- 172.21.15.84:0/2154866623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f66d0 0x7f61e00f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 --2- 172.21.15.84:0/2154866623 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f5430 0x7f61e00f5850 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 -- 172.21.15.84:0/2154866623 >> 172.21.15.84:0/2154866623 conn(0x7f61e00f0b90 msgr2=0x7f61e00f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 -- 172.21.15.84:0/2154866623 shutdown_connections 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.288+0000 7f61dffff700 1 -- 172.21.15.84:0/2154866623 wait complete. 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 Processor -- start 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 -- start start 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 0x7f61e0109ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:12.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00603c0 con 0x7f61e00f66d0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.289+0000 7f61dffff700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0060500 con 0x7f61e00f5430 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:46860/0 (socket says 172.21.15.84:46860) 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 -- 172.21.15.84:0/1438816069 learned_addr learned my addr 172.21.15.84:0/1438816069 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61deffd700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 0x7f61e0109ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 -- 172.21.15.84:0/1438816069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 msgr2=0x7f61e0109ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 0x7f61e0109ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 -- 172.21.15.84:0/1438816069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d0005040 con 0x7f61e00f66d0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61deffd700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 0x7f61e0109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61de7fc700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f61d40026d0 tx=0x7f61d4009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d4009320 con 0x7f61e00f66d0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d400f040 con 0x7f61e00f66d0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e010c580 con 0x7f61e00f66d0 2024-01-18T07:24:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61d40147e0 con 0x7f61e00f66d0 2024-01-18T07:24:12.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.290+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e010cb10 con 0x7f61e00f66d0 2024-01-18T07:24:12.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.291+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0059100 con 0x7f61e00f66d0 2024-01-18T07:24:12.294 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.291+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f61d4009480 con 0x7f61e00f66d0 2024-01-18T07:24:12.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.292+0000 7f61c77fe700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 0x7f61c805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:12.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.292+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1385+0+0 (secure 0 0 0) 0x7f61d40754f0 con 0x7f61e00f66d0 2024-01-18T07:24:12.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.292+0000 7f61deffd700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 0x7f61c805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:12.295 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.292+0000 7f61deffd700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 0x7f61c805e7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f61e0060040 tx=0x7f61d0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:12.296 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.296+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61d40491d0 con 0x7f61e00f66d0 2024-01-18T07:24:12.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:12.475+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f61e00fdd30 con 0x7f61c805c300 2024-01-18T07:24:13.047 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:24:12.752+0000 7f96529d3080 -1 osd.0 0 log_to_monitors {default=true} 2024-01-18T07:24:13.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:10.940126+0000 mgr.smithi084.sihbqg (mgr.14162) 152 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:13.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.477478+0000 mon.smithi084 (mon.0) 270 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:13.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.478675+0000 mon.smithi084 (mon.0) 271 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:13.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.479185+0000 mon.smithi084 (mon.0) 272 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:13.047 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.756766+0000 mon.smithi084 (mon.0) 273 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-18T07:24:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:10.940126+0000 mgr.smithi084.sihbqg (mgr.14162) 152 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.477478+0000 mon.smithi084 (mon.0) 270 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:13.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.478675+0000 mon.smithi084 (mon.0) 271 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:13.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.479185+0000 mon.smithi084 (mon.0) 272 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:13.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.756766+0000 mon.smithi084 (mon.0) 273 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-18T07:24:14.382 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:24:13.931+0000 7f9648f44700 -1 osd.0 0 waiting for initial osdmap 2024-01-18T07:24:14.382 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:24:13.934+0000 7f9644ebf700 -1 osd.0 8 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.476771+0000 mgr.smithi084.sihbqg (mgr.14162) 153 : audit [DBG] from='client.14362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:12.478468+0000 mgr.smithi084.sihbqg (mgr.14162) 154 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.923921+0000 mon.smithi084 (mon.0) 274 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:12.923964+0000 mon.smithi084 (mon.0) 275 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.924049+0000 mon.smithi084 (mon.0) 276 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:14.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:12.924357+0000 mon.smithi084 (mon.0) 277 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:14.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.476771+0000 mgr.smithi084.sihbqg (mgr.14162) 153 : audit [DBG] from='client.14362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:14.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:12.478468+0000 mgr.smithi084.sihbqg (mgr.14162) 154 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:14.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.923921+0000 mon.smithi084 (mon.0) 274 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-18T07:24:14.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:12.923964+0000 mon.smithi084 (mon.0) 275 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-01-18T07:24:14.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.924049+0000 mon.smithi084 (mon.0) 276 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:14.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:12.924357+0000 mon.smithi084 (mon.0) 277 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:12.940444+0000 mgr.smithi084.sihbqg (mgr.14162) 155 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:13.928908+0000 mon.smithi084 (mon.0) 278 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:13.928950+0000 mon.smithi084 (mon.0) 279 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:13.929217+0000 mon.smithi084 (mon.0) 280 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:13.932116+0000 mon.smithi084 (mon.0) 281 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:14.068332+0000 mon.smithi084 (mon.0) 282 : audit [INF] from='client.? 172.21.15.84:0/1606761410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b89774fa-ff6c-49dc-91d4-296a30dbb3a3"}]: dispatch 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:14.072004+0000 mon.smithi084 (mon.0) 283 : cluster [INF] osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] boot 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:14.072048+0000 mon.smithi084 (mon.0) 284 : audit [INF] from='client.? 172.21.15.84:0/1606761410' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b89774fa-ff6c-49dc-91d4-296a30dbb3a3"}]': finished 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:14.072098+0000 mon.smithi084 (mon.0) 285 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:14.072350+0000 mon.smithi084 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:14.072579+0000 mon.smithi084 (mon.0) 287 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:15.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:14.471684+0000 mon.smithi084 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.84:0/461377566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:12.940444+0000 mgr.smithi084.sihbqg (mgr.14162) 155 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:13.928908+0000 mon.smithi084 (mon.0) 278 : audit [INF] from='osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:13.928950+0000 mon.smithi084 (mon.0) 279 : cluster [DBG] osdmap e8: 1 total, 0 up, 1 in 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:13.929217+0000 mon.smithi084 (mon.0) 280 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:13.932116+0000 mon.smithi084 (mon.0) 281 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:14.068332+0000 mon.smithi084 (mon.0) 282 : audit [INF] from='client.? 172.21.15.84:0/1606761410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b89774fa-ff6c-49dc-91d4-296a30dbb3a3"}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:14.072004+0000 mon.smithi084 (mon.0) 283 : cluster [INF] osd.0 [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] boot 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:14.072048+0000 mon.smithi084 (mon.0) 284 : audit [INF] from='client.? 172.21.15.84:0/1606761410' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b89774fa-ff6c-49dc-91d4-296a30dbb3a3"}]': finished 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:14.072098+0000 mon.smithi084 (mon.0) 285 : cluster [DBG] osdmap e9: 2 total, 1 up, 2 in 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:14.072350+0000 mon.smithi084 (mon.0) 286 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:14.072579+0000 mon.smithi084 (mon.0) 287 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:15.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:14.471684+0000 mon.smithi084 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.84:0/461377566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:13.752914+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:13.753011+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:14.940815+0000 mgr.smithi084.sihbqg (mgr.14162) 156 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:15.076969+0000 mon.smithi084 (mon.0) 289 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-01-18T07:24:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:15.077059+0000 mon.smithi084 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:16.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:13.752914+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:16.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:13.753011+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:16.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:14.940815+0000 mgr.smithi084.sihbqg (mgr.14162) 156 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:16.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:15.076969+0000 mon.smithi084 (mon.0) 289 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-01-18T07:24:16.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:15.077059+0000 mon.smithi084 (mon.0) 290 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:16.941060+0000 mgr.smithi084.sihbqg (mgr.14162) 157 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:18.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:16.941060+0000 mgr.smithi084.sihbqg (mgr.14162) 157 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:18.941299+0000 mgr.smithi084.sihbqg (mgr.14162) 158 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:19.824781+0000 mon.smithi084 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-18T07:24:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:19.825366+0000 mon.smithi084 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:19.825846+0000 mgr.smithi084.sihbqg (mgr.14162) 159 : cephadm [INF] Deploying daemon osd.1 on smithi084 2024-01-18T07:24:20.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:18.941299+0000 mgr.smithi084.sihbqg (mgr.14162) 158 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:20.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:19.824781+0000 mon.smithi084 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-18T07:24:20.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:19.825366+0000 mon.smithi084 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:20.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:19.825846+0000 mgr.smithi084.sihbqg (mgr.14162) 159 : cephadm [INF] Deploying daemon osd.1 on smithi084 2024-01-18T07:24:22.101 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 1 on host 'smithi084' 2024-01-18T07:24:22.101 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.098+0000 7f61c77fe700 1 -- 172.21.15.84:0/1438816069 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61e00fdd30 con 0x7f61c805c300 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 msgr2=0x7f61c805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 0x7f61c805e7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f61e0060040 tx=0x7f61d0009b30).stop 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 msgr2=0x7f61e010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f61d40026d0 tx=0x7f61d4009de0).stop 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 shutdown_connections 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f61c805c300 0x7f61c805e7c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f61e00f5430 0x7f61e0109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:22.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 --2- 172.21.15.84:0/1438816069 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f61e00f66d0 0x7f61e010bfd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:22.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.101+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 >> 172.21.15.84:0/1438816069 conn(0x7f61e00f0b90 msgr2=0x7f61e00fa120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:22.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.102+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 shutdown_connections 2024-01-18T07:24:22.103 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:22.102+0000 7f61dffff700 1 -- 172.21.15.84:0/1438816069 wait complete. 2024-01-18T07:24:22.459 DEBUG:teuthology.orchestra.run.smithi084:osd.1> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.1.service 2024-01-18T07:24:22.461 INFO:tasks.cephadm:Deploying osd.2 on smithi084 with /dev/vg_nvme/lv_2... 2024-01-18T07:24:22.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-18T07:24:22.490 INFO:journalctl@ceph.osd.1.smithi084.stdout:-- Logs begin at Thu 2024-01-18 07:09:07 UTC. -- 2024-01-18T07:24:22.824 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:24:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:24:22.565+0000 7f8e73bf7080 -1 Falling back to public interface 2024-01-18T07:24:23.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:20.941561+0000 mgr.smithi084.sihbqg (mgr.14162) 160 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:23.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:22.095174+0000 mon.smithi084 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:23.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:22.099113+0000 mon.smithi084 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:23.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:20.941561+0000 mgr.smithi084.sihbqg (mgr.14162) 160 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:23.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:22.095174+0000 mon.smithi084 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:23.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:22.099113+0000 mon.smithi084 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:23.953 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-18T07:24:23.953 INFO:teuthology.orchestra.run.smithi084.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-18T07:24:23.953 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 10+0 records in 2024-01-18T07:24:23.953 INFO:teuthology.orchestra.run.smithi084.stderr:10+0 records out 2024-01-18T07:24:23.954 INFO:teuthology.orchestra.run.smithi084.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121627 s, 862 MB/s 2024-01-18T07:24:23.954 INFO:teuthology.orchestra.run.smithi084.stderr: stderr: 2024-01-18T07:24:23.954 INFO:teuthology.orchestra.run.smithi084.stderr:--> Zapping successful for: 2024-01-18T07:24:24.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi084:vg_nvme/lv_2 2024-01-18T07:24:25.053 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:22.941901+0000 mgr.smithi084.sihbqg (mgr.14162) 161 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:25.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:22.941901+0000 mgr.smithi084.sihbqg (mgr.14162) 161 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:25.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 -- 172.21.15.84:0/3721169137 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f66e0 msgr2=0x7fd4640f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:25.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 --2- 172.21.15.84:0/3721169137 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f66e0 0x7fd4640f6b40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fd454004660 tx=0x7fd454009b30).stop 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 -- 172.21.15.84:0/3721169137 shutdown_connections 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 --2- 172.21.15.84:0/3721169137 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f66e0 0x7fd4640f6b40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 --2- 172.21.15.84:0/3721169137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f5440 0x7fd4640f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 -- 172.21.15.84:0/3721169137 >> 172.21.15.84:0/3721169137 conn(0x7fd4640f0ba0 msgr2=0x7fd4640f2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 -- 172.21.15.84:0/3721169137 shutdown_connections 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.561+0000 7fd469219700 1 -- 172.21.15.84:0/3721169137 wait complete. 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 Processor -- start 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 -- start start 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f66e0 0x7fd46410bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4640603c0 con 0x7fd4640f5440 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd469219700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd464060500 con 0x7fd4640f66e0 2024-01-18T07:24:25.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40864/0 (socket says 172.21.15.84:40864) 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 -- 172.21.15.84:0/1417889314 learned_addr learned my addr 172.21.15.84:0/1417889314 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd4637fe700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f66e0 0x7fd46410bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 -- 172.21.15.84:0/1417889314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f66e0 msgr2=0x7fd46410bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f66e0 0x7fd46410bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.562+0000 7fd463fff700 1 -- 172.21.15.84:0/1417889314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454005040 con 0x7fd4640f5440 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd463fff700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd44c00c530 tx=0x7fd44c007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44c00f650 con 0x7fd4640f5440 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46410c580 con 0x7fd4640f5440 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd44c00fc60 con 0x7fd4640f5440 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd44c012d20 con 0x7fd4640f5440 2024-01-18T07:24:25.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.563+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46410cb10 con 0x7fd4640f5440 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.564+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd464059100 con 0x7fd4640f5440 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.564+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd44c019030 con 0x7fd4640f5440 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.564+0000 7fd460ff9700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 0x7fd45005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.564+0000 7fd4637fe700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 0x7fd45005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.565+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1997+0+0 (secure 0 0 0) 0x7fd44c074e30 con 0x7fd4640f5440 2024-01-18T07:24:25.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.565+0000 7fd4637fe700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 0x7fd45005e7c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd454009d60 tx=0x7fd454009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:25.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.568+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd44c048880 con 0x7fd4640f5440 2024-01-18T07:24:25.745 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:25.745+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4640fdd80 con 0x7fd45005c300 2024-01-18T07:24:26.082 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:24:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:24:25.822+0000 7f8e73bf7080 -1 osd.1 0 log_to_monitors {default=true} 2024-01-18T07:24:26.083 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.747311+0000 mon.smithi084 (mon.0) 295 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:26.083 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.748499+0000 mon.smithi084 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:26.083 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.748988+0000 mon.smithi084 (mon.0) 297 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:26.083 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.825532+0000 mon.smithi084 (mon.0) 298 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-18T07:24:26.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.747311+0000 mon.smithi084 (mon.0) 295 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:26.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.748499+0000 mon.smithi084 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:26.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.748988+0000 mon.smithi084 (mon.0) 297 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:26.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.825532+0000 mon.smithi084 (mon.0) 298 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-18T07:24:27.044 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:24:26.928+0000 7f8e6a168700 -1 osd.1 0 waiting for initial osdmap 2024-01-18T07:24:27.044 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:24:26.935+0000 7f8e640df700 -1 osd.1 12 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:24:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:24.942134+0000 mgr.smithi084.sihbqg (mgr.14162) 162 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:27.044 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.746576+0000 mgr.smithi084.sihbqg (mgr.14162) 163 : audit [DBG] from='client.14380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:25.748325+0000 mgr.smithi084.sihbqg (mgr.14162) 164 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.923789+0000 mon.smithi084 (mon.0) 299 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-18T07:24:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:25.923879+0000 mon.smithi084 (mon.0) 300 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-01-18T07:24:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.923973+0000 mon.smithi084 (mon.0) 301 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:27.045 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:25.924325+0000 mon.smithi084 (mon.0) 302 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:24.942134+0000 mgr.smithi084.sihbqg (mgr.14162) 162 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.746576+0000 mgr.smithi084.sihbqg (mgr.14162) 163 : audit [DBG] from='client.14380 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:25.748325+0000 mgr.smithi084.sihbqg (mgr.14162) 164 : cephadm [INF] Applying service osd.None on host smithi084... 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.923789+0000 mon.smithi084 (mon.0) 299 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:25.923879+0000 mon.smithi084 (mon.0) 300 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.923973+0000 mon.smithi084 (mon.0) 301 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:25.924325+0000 mon.smithi084 (mon.0) 302 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:26.925682+0000 mon.smithi084 (mon.0) 303 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:26.925763+0000 mon.smithi084 (mon.0) 304 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-01-18T07:24:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:26.926194+0000 mon.smithi084 (mon.0) 305 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:26.930190+0000 mon.smithi084 (mon.0) 306 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:27.728749+0000 mon.smithi084 (mon.0) 307 : audit [INF] from='client.? 172.21.15.84:0/3730661133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0543c962-6df6-4a1b-8c7a-1335c7c5f501"}]: dispatch 2024-01-18T07:24:28.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:27.735007+0000 mon.smithi084 (mon.0) 308 : cluster [INF] osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] boot 2024-01-18T07:24:28.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:27.735051+0000 mon.smithi084 (mon.0) 309 : audit [INF] from='client.? 172.21.15.84:0/3730661133' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0543c962-6df6-4a1b-8c7a-1335c7c5f501"}]': finished 2024-01-18T07:24:28.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:27.735092+0000 mon.smithi084 (mon.0) 310 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2024-01-18T07:24:28.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:27.735393+0000 mon.smithi084 (mon.0) 311 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:27.735589+0000 mon.smithi084 (mon.0) 312 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:26.925682+0000 mon.smithi084 (mon.0) 303 : audit [INF] from='osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:26.925763+0000 mon.smithi084 (mon.0) 304 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:26.926194+0000 mon.smithi084 (mon.0) 305 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:26.930190+0000 mon.smithi084 (mon.0) 306 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:27.728749+0000 mon.smithi084 (mon.0) 307 : audit [INF] from='client.? 172.21.15.84:0/3730661133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0543c962-6df6-4a1b-8c7a-1335c7c5f501"}]: dispatch 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:27.735007+0000 mon.smithi084 (mon.0) 308 : cluster [INF] osd.1 [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] boot 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:27.735051+0000 mon.smithi084 (mon.0) 309 : audit [INF] from='client.? 172.21.15.84:0/3730661133' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0543c962-6df6-4a1b-8c7a-1335c7c5f501"}]': finished 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:27.735092+0000 mon.smithi084 (mon.0) 310 : cluster [DBG] osdmap e13: 3 total, 2 up, 3 in 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:27.735393+0000 mon.smithi084 (mon.0) 311 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-18T07:24:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:27.735589+0000 mon.smithi084 (mon.0) 312 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:29.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:26.797461+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:29.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:26.797562+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:29.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:26.942381+0000 mgr.smithi084.sihbqg (mgr.14162) 165 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:29.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:28.132922+0000 mon.smithi084 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.84:0/863729670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:29.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:28.737228+0000 mon.smithi084 (mon.0) 314 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2024-01-18T07:24:29.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:28.737349+0000 mon.smithi084 (mon.0) 315 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:26.797461+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:29.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:26.797562+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:29.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:26.942381+0000 mgr.smithi084.sihbqg (mgr.14162) 165 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2024-01-18T07:24:29.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:28.132922+0000 mon.smithi084 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.84:0/863729670' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:24:29.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:28.737228+0000 mon.smithi084 (mon.0) 314 : cluster [DBG] osdmap e14: 3 total, 2 up, 3 in 2024-01-18T07:24:29.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:28.737349+0000 mon.smithi084 (mon.0) 315 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:30.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:28.942649+0000 mgr.smithi084.sihbqg (mgr.14162) 166 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:30.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:28.942649+0000 mgr.smithi084.sihbqg (mgr.14162) 166 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:33.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:30.942956+0000 mgr.smithi084.sihbqg (mgr.14162) 167 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:33.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:30.942956+0000 mgr.smithi084.sihbqg (mgr.14162) 167 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:34.325 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:33.572836+0000 mon.smithi084 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-18T07:24:34.326 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:33.573411+0000 mon.smithi084 (mon.0) 317 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:33.572836+0000 mon.smithi084 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-18T07:24:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:33.573411+0000 mon.smithi084 (mon.0) 317 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:34.965 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:32.943206+0000 mgr.smithi084.sihbqg (mgr.14162) 168 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:34.965 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:33.573904+0000 mgr.smithi084.sihbqg (mgr.14162) 169 : cephadm [INF] Deploying daemon osd.2 on smithi084 2024-01-18T07:24:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:32.943206+0000 mgr.smithi084.sihbqg (mgr.14162) 168 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:33.573904+0000 mgr.smithi084.sihbqg (mgr.14162) 169 : cephadm [INF] Deploying daemon osd.2 on smithi084 2024-01-18T07:24:36.009 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 2 on host 'smithi084' 2024-01-18T07:24:36.009 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.007+0000 7fd460ff9700 1 -- 172.21.15.84:0/1417889314 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd454013410 con 0x7fd45005c300 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 msgr2=0x7fd45005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 0x7fd45005e7c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd454009d60 tx=0x7fd454009b30).stop 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 msgr2=0x7fd464109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd44c00c530 tx=0x7fd44c007e10).stop 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 shutdown_connections 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd4640f5440 0x7fd464109ad0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd45005c300 0x7fd45005e7c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 --2- 172.21.15.84:0/1417889314 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd4640f66e0 0x7fd46410bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 >> 172.21.15.84:0/1417889314 conn(0x7fd4640f0ba0 msgr2=0x7fd4640fa130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 shutdown_connections 2024-01-18T07:24:36.011 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:24:36.010+0000 7fd469219700 1 -- 172.21.15.84:0/1417889314 wait complete. 2024-01-18T07:24:36.216 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:35.035692+0000 mon.smithi084 (mon.0) 318 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:24:36.217 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:35.042863+0000 mon.smithi084 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:24:36.401 DEBUG:teuthology.orchestra.run.smithi084:osd.2> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.2.service 2024-01-18T07:24:36.403 INFO:tasks.cephadm:Deploying osd.3 on smithi106 with /dev/vg_nvme/lv_4... 2024-01-18T07:24:36.403 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-18T07:24:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:35.035692+0000 mon.smithi084 (mon.0) 318 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:24:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:35.042863+0000 mon.smithi084 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:24:36.432 INFO:journalctl@ceph.osd.2.smithi084.stdout:-- Logs begin at Thu 2024-01-18 07:09:07 UTC. -- 2024-01-18T07:24:36.882 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:24:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:24:36.468+0000 7f952425e080 -1 Falling back to public interface 2024-01-18T07:24:37.034 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:34.943469+0000 mgr.smithi084.sihbqg (mgr.14162) 170 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:37.035 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:36.002355+0000 mon.smithi084 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:37.035 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:36.007917+0000 mon.smithi084 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:34.943469+0000 mgr.smithi084.sihbqg (mgr.14162) 170 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:36.002355+0000 mon.smithi084 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:36.007917+0000 mon.smithi084 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:24:37.744 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-18T07:24:37.744 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-18T07:24:37.744 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-01-18T07:24:37.744 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-01-18T07:24:37.745 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110331 s, 950 MB/s 2024-01-18T07:24:37.745 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-01-18T07:24:38.122 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-01-18T07:24:39.068 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:36.943837+0000 mgr.smithi084.sihbqg (mgr.14162) 171 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 -- 172.21.15.106:0/2120118778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 msgr2=0x7fe2740f59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 --2- 172.21.15.106:0/2120118778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe2740f59f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fe268004660 tx=0x7fe268009b30).stop 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 -- 172.21.15.106:0/2120118778 shutdown_connections 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 --2- 172.21.15.106:0/2120118778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 0x7fe2740f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 --2- 172.21.15.106:0/2120118778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe2740f59f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 -- 172.21.15.106:0/2120118778 >> 172.21.15.106:0/2120118778 conn(0x7fe2740f0d30 msgr2=0x7fe2740f3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 -- 172.21.15.106:0/2120118778 shutdown_connections 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.248+0000 7fe27aea8700 1 -- 172.21.15.106:0/2120118778 wait complete. 2024-01-18T07:24:39.249 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 Processor -- start 2024-01-18T07:24:39.250 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 -- start start 2024-01-18T07:24:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 0x7fe27410c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:39.251 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740604e0 con 0x7fe2740f6870 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe27aea8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe274060620 con 0x7fe2740f55d0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:34386/0 (socket says 172.21.15.106:34386) 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 -- 172.21.15.106:0/3836215186 learned_addr learned my addr 172.21.15.106:0/3836215186 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe273fff700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 0x7fe27410c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 -- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 msgr2=0x7fe27410c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 0x7fe27410c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.249+0000 7fe278c44700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe268005040 con 0x7fe2740f55d0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe278c44700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fe268000c00 tx=0x7fe26800b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe268019070 con 0x7fe2740f55d0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe268007e70 con 0x7fe2740f55d0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe268014d40 con 0x7fe2740f55d0 2024-01-18T07:24:39.252 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27410c680 con 0x7fe2740f55d0 2024-01-18T07:24:39.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.250+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27410cb90 con 0x7fe2740f55d0 2024-01-18T07:24:39.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.251+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe274045ad0 con 0x7fe2740f55d0 2024-01-18T07:24:39.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.251+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe268025070 con 0x7fe2740f55d0 2024-01-18T07:24:39.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.252+0000 7fe2717fa700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 0x7fe26405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:24:39.253 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.252+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2429+0+0 (secure 0 0 0) 0x7fe268076650 con 0x7fe2740f55d0 2024-01-18T07:24:39.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.252+0000 7fe273fff700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 0x7fe26405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:24:39.254 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.252+0000 7fe273fff700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 0x7fe26405e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe260009fd0 tx=0x7fe26000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:24:39.255 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.255+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe268049f90 con 0x7fe2740f55d0 2024-01-18T07:24:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:36.943837+0000 mgr.smithi084.sihbqg (mgr.14162) 171 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:39.429 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:24:39.428+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe274057f10 con 0x7fe26405c2b0 2024-01-18T07:24:40.076 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.431114+0000 mon.smithi084 (mon.0) 322 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:40.076 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.432570+0000 mon.smithi084 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:40.076 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.433111+0000 mon.smithi084 (mon.0) 324 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:40.076 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.904992+0000 mon.smithi084 (mon.0) 325 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-18T07:24:40.382 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:24:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:24:39.901+0000 7f952425e080 -1 osd.2 0 log_to_monitors {default=true} 2024-01-18T07:24:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.431114+0000 mon.smithi084 (mon.0) 322 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:24:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.432570+0000 mon.smithi084 (mon.0) 323 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:24:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.433111+0000 mon.smithi084 (mon.0) 324 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:24:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.904992+0000 mon.smithi084 (mon.0) 325 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-18T07:24:41.382 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:24:40.941+0000 7f951a7cf700 -1 osd.2 0 waiting for initial osdmap 2024-01-18T07:24:41.382 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:24:40.947+0000 7f9516f4b700 -1 osd.2 16 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:24:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:38.944332+0000 mgr.smithi084.sihbqg (mgr.14162) 172 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.430242+0000 mgr.smithi084.sihbqg (mgr.14162) 173 : audit [DBG] from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:24:39.432362+0000 mgr.smithi084.sihbqg (mgr.14162) 174 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.937473+0000 mon.smithi084 (mon.0) 326 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:39.937539+0000 mon.smithi084 (mon.0) 327 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.937842+0000 mon.smithi084 (mon.0) 328 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:41.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:39.938232+0000 mon.smithi084 (mon.0) 329 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:38.944332+0000 mgr.smithi084.sihbqg (mgr.14162) 172 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.430242+0000 mgr.smithi084.sihbqg (mgr.14162) 173 : audit [DBG] from='client.24123 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:24:41.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:24:39.432362+0000 mgr.smithi084.sihbqg (mgr.14162) 174 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:24:41.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.937473+0000 mon.smithi084 (mon.0) 326 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-18T07:24:41.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:39.937539+0000 mon.smithi084 (mon.0) 327 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-01-18T07:24:41.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.937842+0000 mon.smithi084 (mon.0) 328 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:41.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:39.938232+0000 mon.smithi084 (mon.0) 329 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-01-18T07:24:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:40.938720+0000 mon.smithi084 (mon.0) 330 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:40.938797+0000 mon.smithi084 (mon.0) 331 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-01-18T07:24:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:40.939221+0000 mon.smithi084 (mon.0) 332 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:40.943356+0000 mon.smithi084 (mon.0) 333 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:40.944668+0000 mgr.smithi084.sihbqg (mgr.14162) 175 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:40.938720+0000 mon.smithi084 (mon.0) 330 : audit [INF] from='osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-01-18T07:24:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:40.938797+0000 mon.smithi084 (mon.0) 331 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-01-18T07:24:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:40.939221+0000 mon.smithi084 (mon.0) 332 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:42.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:40.943356+0000 mon.smithi084 (mon.0) 333 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:42.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:40.944668+0000 mgr.smithi084.sihbqg (mgr.14162) 175 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2024-01-18T07:24:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:40.886275+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:40.886389+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:41.944339+0000 mon.smithi084 (mon.0) 334 : cluster [INF] osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] boot 2024-01-18T07:24:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:41.944381+0000 mon.smithi084 (mon.0) 335 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2024-01-18T07:24:43.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:24:41.944813+0000 mon.smithi084 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:40.886275+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:24:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:40.886389+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:24:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:41.944339+0000 mon.smithi084 (mon.0) 334 : cluster [INF] osd.2 [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] boot 2024-01-18T07:24:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:41.944381+0000 mon.smithi084 (mon.0) 335 : cluster [DBG] osdmap e17: 3 total, 3 up, 3 in 2024-01-18T07:24:43.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:24:41.944813+0000 mon.smithi084 (mon.0) 336 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-18T07:24:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:42.945063+0000 mgr.smithi084.sihbqg (mgr.14162) 176 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:42.954333+0000 mon.smithi084 (mon.0) 337 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-01-18T07:24:44.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:42.945063+0000 mgr.smithi084.sihbqg (mgr.14162) 176 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:44.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:42.954333+0000 mon.smithi084 (mon.0) 337 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-01-18T07:24:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:44.945431+0000 mgr.smithi084.sihbqg (mgr.14162) 177 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:44.945431+0000 mgr.smithi084.sihbqg (mgr.14162) 177 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:46.945840+0000 mgr.smithi084.sihbqg (mgr.14162) 178 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:46.945840+0000 mgr.smithi084.sihbqg (mgr.14162) 178 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:48.946159+0000 mgr.smithi084.sihbqg (mgr.14162) 179 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:48.946159+0000 mgr.smithi084.sihbqg (mgr.14162) 179 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:50.946578+0000 mgr.smithi084.sihbqg (mgr.14162) 180 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:50.946578+0000 mgr.smithi084.sihbqg (mgr.14162) 180 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:52.947004+0000 mgr.smithi084.sihbqg (mgr.14162) 181 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:52.947004+0000 mgr.smithi084.sihbqg (mgr.14162) 181 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:54.947406+0000 mgr.smithi084.sihbqg (mgr.14162) 182 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:54.947406+0000 mgr.smithi084.sihbqg (mgr.14162) 182 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:56.947771+0000 mgr.smithi084.sihbqg (mgr.14162) 183 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:24:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:56.947771+0000 mgr.smithi084.sihbqg (mgr.14162) 183 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:24:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:24:58.948202+0000 mgr.smithi084.sihbqg (mgr.14162) 184 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:24:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:24:58.948202+0000 mgr.smithi084.sihbqg (mgr.14162) 184 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:00.948608+0000 mgr.smithi084.sihbqg (mgr.14162) 185 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:00.948608+0000 mgr.smithi084.sihbqg (mgr.14162) 185 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:02.949087+0000 mgr.smithi084.sihbqg (mgr.14162) 186 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:02.949087+0000 mgr.smithi084.sihbqg (mgr.14162) 186 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:04.949433+0000 mgr.smithi084.sihbqg (mgr.14162) 187 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:04.949433+0000 mgr.smithi084.sihbqg (mgr.14162) 187 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:06.949717+0000 mgr.smithi084.sihbqg (mgr.14162) 188 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:06.949717+0000 mgr.smithi084.sihbqg (mgr.14162) 188 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:08.950050+0000 mgr.smithi084.sihbqg (mgr.14162) 189 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:08.950050+0000 mgr.smithi084.sihbqg (mgr.14162) 189 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:10.950379+0000 mgr.smithi084.sihbqg (mgr.14162) 190 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:12.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:10.950379+0000 mgr.smithi084.sihbqg (mgr.14162) 190 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:14.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:12.950795+0000 mgr.smithi084.sihbqg (mgr.14162) 191 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:14.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:12.950795+0000 mgr.smithi084.sihbqg (mgr.14162) 191 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:14.951120+0000 mgr.smithi084.sihbqg (mgr.14162) 192 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:14.951120+0000 mgr.smithi084.sihbqg (mgr.14162) 192 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:16.951418+0000 mgr.smithi084.sihbqg (mgr.14162) 193 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:16.951418+0000 mgr.smithi084.sihbqg (mgr.14162) 193 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:18.951874+0000 mgr.smithi084.sihbqg (mgr.14162) 194 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:18.951874+0000 mgr.smithi084.sihbqg (mgr.14162) 194 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:20.952296+0000 mgr.smithi084.sihbqg (mgr.14162) 195 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:20.952296+0000 mgr.smithi084.sihbqg (mgr.14162) 195 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:22.952645+0000 mgr.smithi084.sihbqg (mgr.14162) 196 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:22.952645+0000 mgr.smithi084.sihbqg (mgr.14162) 196 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:24.953102+0000 mgr.smithi084.sihbqg (mgr.14162) 197 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:24.953102+0000 mgr.smithi084.sihbqg (mgr.14162) 197 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:26.953473+0000 mgr.smithi084.sihbqg (mgr.14162) 198 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:26.953473+0000 mgr.smithi084.sihbqg (mgr.14162) 198 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:28.953832+0000 mgr.smithi084.sihbqg (mgr.14162) 199 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:28.953832+0000 mgr.smithi084.sihbqg (mgr.14162) 199 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:30.954174+0000 mgr.smithi084.sihbqg (mgr.14162) 200 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:30.954174+0000 mgr.smithi084.sihbqg (mgr.14162) 200 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:32.954480+0000 mgr.smithi084.sihbqg (mgr.14162) 201 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:32.954480+0000 mgr.smithi084.sihbqg (mgr.14162) 201 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:35.037692+0000 mon.smithi084 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:25:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:35.044877+0000 mon.smithi084 (mon.0) 339 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:25:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:35.037692+0000 mon.smithi084 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:25:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:35.044877+0000 mon.smithi084 (mon.0) 339 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:25:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:34.954907+0000 mgr.smithi084.sihbqg (mgr.14162) 202 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:34.954907+0000 mgr.smithi084.sihbqg (mgr.14162) 202 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:36.955174+0000 mgr.smithi084.sihbqg (mgr.14162) 203 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:38.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:36.955174+0000 mgr.smithi084.sihbqg (mgr.14162) 203 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:38.955492+0000 mgr.smithi084.sihbqg (mgr.14162) 204 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:40.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:38.955492+0000 mgr.smithi084.sihbqg (mgr.14162) 204 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:40.955987+0000 mgr.smithi084.sihbqg (mgr.14162) 205 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:40.955987+0000 mgr.smithi084.sihbqg (mgr.14162) 205 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:42.956456+0000 mgr.smithi084.sihbqg (mgr.14162) 206 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:42.956456+0000 mgr.smithi084.sihbqg (mgr.14162) 206 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:44.956940+0000 mgr.smithi084.sihbqg (mgr.14162) 207 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:44.956940+0000 mgr.smithi084.sihbqg (mgr.14162) 207 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:25:46.204215+0000 mgr.smithi084.sihbqg (mgr.14162) 208 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi106 ... 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Verifying port 9100 ... 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:25:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi106 ... 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Verifying port 9100 ... 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:25:48.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:25:46.207489+0000 mgr.smithi084.sihbqg (mgr.14162) 209 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:47.044757+0000 mon.smithi106 (mon.1) 2 : audit [INF] from='client.? 172.21.15.106:0/2693813633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]: dispatch 2024-01-18T07:25:48.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:47.045341+0000 mon.smithi084 (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]: dispatch 2024-01-18T07:25:48.387 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:47.051521+0000 mon.smithi084 (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]': finished 2024-01-18T07:25:48.387 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:47.051607+0000 mon.smithi084 (mon.0) 342 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2024-01-18T07:25:48.387 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:47.051706+0000 mon.smithi084 (mon.0) 343 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:25:48.387 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:25:47.469038+0000 mon.smithi106 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.106:0/946460788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:25:48.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:25:46.204215+0000 mgr.smithi084.sihbqg (mgr.14162) 208 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi106 ... 2024-01-18T07:25:48.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Verifying port 9100 ... 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T07:25:48.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi106 ... 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Verifying port 9100 ... 2024-01-18T07:25:48.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-18T07:25:48.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl start ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:25:46.207489+0000 mgr.smithi084.sihbqg (mgr.14162) 209 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:47.044757+0000 mon.smithi106 (mon.1) 2 : audit [INF] from='client.? 172.21.15.106:0/2693813633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]: dispatch 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:47.045341+0000 mon.smithi084 (mon.0) 340 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]: dispatch 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:47.051521+0000 mon.smithi084 (mon.0) 341 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cbac515c-d096-47c9-ada0-a838b722770e"}]': finished 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:47.051607+0000 mon.smithi084 (mon.0) 342 : cluster [DBG] osdmap e19: 4 total, 3 up, 4 in 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:47.051706+0000 mon.smithi084 (mon.0) 343 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:25:48.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:25:47.469038+0000 mon.smithi106 (mon.1) 3 : audit [DBG] from='client.? 172.21.15.106:0/946460788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:25:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:46.957303+0000 mgr.smithi084.sihbqg (mgr.14162) 210 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:46.957303+0000 mgr.smithi084.sihbqg (mgr.14162) 210 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:48.957702+0000 mgr.smithi084.sihbqg (mgr.14162) 211 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:50.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:48.957702+0000 mgr.smithi084.sihbqg (mgr.14162) 211 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:50.958094+0000 mgr.smithi084.sihbqg (mgr.14162) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:50.958094+0000 mgr.smithi084.sihbqg (mgr.14162) 212 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:52.958623+0000 mgr.smithi084.sihbqg (mgr.14162) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:52.958623+0000 mgr.smithi084.sihbqg (mgr.14162) 213 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:54.959010+0000 mgr.smithi084.sihbqg (mgr.14162) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:54.959010+0000 mgr.smithi084.sihbqg (mgr.14162) 214 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:25:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:56.959455+0000 mgr.smithi084.sihbqg (mgr.14162) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:25:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:25:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:56.959455+0000 mgr.smithi084.sihbqg (mgr.14162) 215 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:25:58.959829+0000 mgr.smithi084.sihbqg (mgr.14162) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:25:58.959829+0000 mgr.smithi084.sihbqg (mgr.14162) 216 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:00.960175+0000 mgr.smithi084.sihbqg (mgr.14162) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:00.960175+0000 mgr.smithi084.sihbqg (mgr.14162) 217 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:04.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:02.960510+0000 mgr.smithi084.sihbqg (mgr.14162) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:04.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:02.960510+0000 mgr.smithi084.sihbqg (mgr.14162) 218 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:04.961011+0000 mgr.smithi084.sihbqg (mgr.14162) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:04.961011+0000 mgr.smithi084.sihbqg (mgr.14162) 219 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:06.961507+0000 mgr.smithi084.sihbqg (mgr.14162) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:06.961507+0000 mgr.smithi084.sihbqg (mgr.14162) 220 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:08.961858+0000 mgr.smithi084.sihbqg (mgr.14162) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:08.961858+0000 mgr.smithi084.sihbqg (mgr.14162) 221 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:10.962128+0000 mgr.smithi084.sihbqg (mgr.14162) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:10.962128+0000 mgr.smithi084.sihbqg (mgr.14162) 222 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:12.962523+0000 mgr.smithi084.sihbqg (mgr.14162) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:12.962523+0000 mgr.smithi084.sihbqg (mgr.14162) 223 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:14.963015+0000 mgr.smithi084.sihbqg (mgr.14162) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:14.963015+0000 mgr.smithi084.sihbqg (mgr.14162) 224 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:16.963437+0000 mgr.smithi084.sihbqg (mgr.14162) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:16.963437+0000 mgr.smithi084.sihbqg (mgr.14162) 225 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:18.963924+0000 mgr.smithi084.sihbqg (mgr.14162) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:18.963924+0000 mgr.smithi084.sihbqg (mgr.14162) 226 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:20.964430+0000 mgr.smithi084.sihbqg (mgr.14162) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:20.964430+0000 mgr.smithi084.sihbqg (mgr.14162) 227 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:22.964830+0000 mgr.smithi084.sihbqg (mgr.14162) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:22.964830+0000 mgr.smithi084.sihbqg (mgr.14162) 228 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:24.965256+0000 mgr.smithi084.sihbqg (mgr.14162) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:24.965256+0000 mgr.smithi084.sihbqg (mgr.14162) 229 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:26.965741+0000 mgr.smithi084.sihbqg (mgr.14162) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:26.965741+0000 mgr.smithi084.sihbqg (mgr.14162) 230 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:31.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:28.966169+0000 mgr.smithi084.sihbqg (mgr.14162) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:28.966169+0000 mgr.smithi084.sihbqg (mgr.14162) 231 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:30.966534+0000 mgr.smithi084.sihbqg (mgr.14162) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:30.966534+0000 mgr.smithi084.sihbqg (mgr.14162) 232 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:32.966881+0000 mgr.smithi084.sihbqg (mgr.14162) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:32.966881+0000 mgr.smithi084.sihbqg (mgr.14162) 233 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:26:35.038923+0000 mon.smithi084 (mon.0) 344 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:26:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:26:35.045964+0000 mon.smithi084 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:26:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:26:35.038923+0000 mon.smithi084 (mon.0) 344 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:26:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:26:35.045964+0000 mon.smithi084 (mon.0) 345 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:26:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:34.967326+0000 mgr.smithi084.sihbqg (mgr.14162) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:34.967326+0000 mgr.smithi084.sihbqg (mgr.14162) 234 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:36.967753+0000 mgr.smithi084.sihbqg (mgr.14162) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:38.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:36.967753+0000 mgr.smithi084.sihbqg (mgr.14162) 235 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:38.968057+0000 mgr.smithi084.sihbqg (mgr.14162) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:40.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:38.968057+0000 mgr.smithi084.sihbqg (mgr.14162) 236 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:40.968502+0000 mgr.smithi084.sihbqg (mgr.14162) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:40.968502+0000 mgr.smithi084.sihbqg (mgr.14162) 237 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:42.969032+0000 mgr.smithi084.sihbqg (mgr.14162) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:42.969032+0000 mgr.smithi084.sihbqg (mgr.14162) 238 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:44.969432+0000 mgr.smithi084.sihbqg (mgr.14162) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:44.969432+0000 mgr.smithi084.sihbqg (mgr.14162) 239 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:46.969784+0000 mgr.smithi084.sihbqg (mgr.14162) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:46.969784+0000 mgr.smithi084.sihbqg (mgr.14162) 240 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:48.970141+0000 mgr.smithi084.sihbqg (mgr.14162) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:48.970141+0000 mgr.smithi084.sihbqg (mgr.14162) 241 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:52.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:50.970518+0000 mgr.smithi084.sihbqg (mgr.14162) 242 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:52.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:50.970518+0000 mgr.smithi084.sihbqg (mgr.14162) 242 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:54.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:52.970898+0000 mgr.smithi084.sihbqg (mgr.14162) 243 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:54.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:52.970898+0000 mgr.smithi084.sihbqg (mgr.14162) 243 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:54.971269+0000 mgr.smithi084.sihbqg (mgr.14162) 244 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:54.971269+0000 mgr.smithi084.sihbqg (mgr.14162) 244 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:26:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:56.971612+0000 mgr.smithi084.sihbqg (mgr.14162) 245 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:26:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:26:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:56.971612+0000 mgr.smithi084.sihbqg (mgr.14162) 245 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:26:58.972086+0000 mgr.smithi084.sihbqg (mgr.14162) 246 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:26:58.972086+0000 mgr.smithi084.sihbqg (mgr.14162) 246 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:00.972563+0000 mgr.smithi084.sihbqg (mgr.14162) 247 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:00.972563+0000 mgr.smithi084.sihbqg (mgr.14162) 247 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:02.973011+0000 mgr.smithi084.sihbqg (mgr.14162) 248 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:02.973011+0000 mgr.smithi084.sihbqg (mgr.14162) 248 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:07.228 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:04.973421+0000 mgr.smithi084.sihbqg (mgr.14162) 249 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:04.973421+0000 mgr.smithi084.sihbqg (mgr.14162) 249 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:06.973739+0000 mgr.smithi084.sihbqg (mgr.14162) 250 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:06.973739+0000 mgr.smithi084.sihbqg (mgr.14162) 250 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:08.974182+0000 mgr.smithi084.sihbqg (mgr.14162) 251 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:08.974182+0000 mgr.smithi084.sihbqg (mgr.14162) 251 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:10.974535+0000 mgr.smithi084.sihbqg (mgr.14162) 252 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:10.974535+0000 mgr.smithi084.sihbqg (mgr.14162) 252 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:12.974918+0000 mgr.smithi084.sihbqg (mgr.14162) 253 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:12.974918+0000 mgr.smithi084.sihbqg (mgr.14162) 253 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:14.975149+0000 mgr.smithi084.sihbqg (mgr.14162) 254 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:14.975149+0000 mgr.smithi084.sihbqg (mgr.14162) 254 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:16.975564+0000 mgr.smithi084.sihbqg (mgr.14162) 255 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:16.975564+0000 mgr.smithi084.sihbqg (mgr.14162) 255 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:18.975903+0000 mgr.smithi084.sihbqg (mgr.14162) 256 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:18.975903+0000 mgr.smithi084.sihbqg (mgr.14162) 256 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:20.976315+0000 mgr.smithi084.sihbqg (mgr.14162) 257 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:20.976315+0000 mgr.smithi084.sihbqg (mgr.14162) 257 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:24.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:22.976756+0000 mgr.smithi084.sihbqg (mgr.14162) 258 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:24.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:22.976756+0000 mgr.smithi084.sihbqg (mgr.14162) 258 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:24.977051+0000 mgr.smithi084.sihbqg (mgr.14162) 259 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:24.977051+0000 mgr.smithi084.sihbqg (mgr.14162) 259 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:26.977478+0000 mgr.smithi084.sihbqg (mgr.14162) 260 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:26.977478+0000 mgr.smithi084.sihbqg (mgr.14162) 260 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:28.977771+0000 mgr.smithi084.sihbqg (mgr.14162) 261 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:28.977771+0000 mgr.smithi084.sihbqg (mgr.14162) 261 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:30.978195+0000 mgr.smithi084.sihbqg (mgr.14162) 262 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:30.978195+0000 mgr.smithi084.sihbqg (mgr.14162) 262 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:32.978502+0000 mgr.smithi084.sihbqg (mgr.14162) 263 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:32.978502+0000 mgr.smithi084.sihbqg (mgr.14162) 263 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:34.978928+0000 mgr.smithi084.sihbqg (mgr.14162) 264 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:27:35.040180+0000 mon.smithi084 (mon.0) 346 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:27:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:27:35.047423+0000 mon.smithi084 (mon.0) 347 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:27:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:34.978928+0000 mgr.smithi084.sihbqg (mgr.14162) 264 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:27:35.040180+0000 mon.smithi084 (mon.0) 346 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:27:36.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:27:35.047423+0000 mon.smithi084 (mon.0) 347 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:27:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:36.979260+0000 mgr.smithi084.sihbqg (mgr.14162) 265 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:38.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:36.979260+0000 mgr.smithi084.sihbqg (mgr.14162) 265 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:39.272 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:27:39.270+0000 7fe25ffff700 1 -- 172.21.15.106:0/3836215186 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe24c0031e0 con 0x7fe2740f55d0 2024-01-18T07:27:39.273 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:27:39.270+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mon.1 v2:172.21.15.106:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7fe24c0031e0 con 0x7fe2740f55d0 2024-01-18T07:27:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:38.979639+0000 mgr.smithi084.sihbqg (mgr.14162) 266 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:40.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:38.979639+0000 mgr.smithi084.sihbqg (mgr.14162) 266 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:40.980058+0000 mgr.smithi084.sihbqg (mgr.14162) 267 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:40.980058+0000 mgr.smithi084.sihbqg (mgr.14162) 267 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:45.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:42.980484+0000 mgr.smithi084.sihbqg (mgr.14162) 268 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:42.980484+0000 mgr.smithi084.sihbqg (mgr.14162) 268 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:44.980766+0000 mgr.smithi084.sihbqg (mgr.14162) 269 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:44.980766+0000 mgr.smithi084.sihbqg (mgr.14162) 269 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:46.981176+0000 mgr.smithi084.sihbqg (mgr.14162) 270 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:46.981176+0000 mgr.smithi084.sihbqg (mgr.14162) 270 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:48.981615+0000 mgr.smithi084.sihbqg (mgr.14162) 271 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:50.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:48.981615+0000 mgr.smithi084.sihbqg (mgr.14162) 271 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:50.981926+0000 mgr.smithi084.sihbqg (mgr.14162) 272 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:50.981926+0000 mgr.smithi084.sihbqg (mgr.14162) 272 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:52.982214+0000 mgr.smithi084.sihbqg (mgr.14162) 273 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:52.982214+0000 mgr.smithi084.sihbqg (mgr.14162) 273 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:54.982641+0000 mgr.smithi084.sihbqg (mgr.14162) 274 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:54.982641+0000 mgr.smithi084.sihbqg (mgr.14162) 274 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:27:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:56.982996+0000 mgr.smithi084.sihbqg (mgr.14162) 275 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:27:58.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:27:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:56.982996+0000 mgr.smithi084.sihbqg (mgr.14162) 275 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:27:58.983351+0000 mgr.smithi084.sihbqg (mgr.14162) 276 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:27:58.983351+0000 mgr.smithi084.sihbqg (mgr.14162) 276 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:00.983658+0000 mgr.smithi084.sihbqg (mgr.14162) 277 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:00.983658+0000 mgr.smithi084.sihbqg (mgr.14162) 277 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:02.984062+0000 mgr.smithi084.sihbqg (mgr.14162) 278 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:02.984062+0000 mgr.smithi084.sihbqg (mgr.14162) 278 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:04.984397+0000 mgr.smithi084.sihbqg (mgr.14162) 279 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:04.984397+0000 mgr.smithi084.sihbqg (mgr.14162) 279 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:08.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:06.984735+0000 mgr.smithi084.sihbqg (mgr.14162) 280 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:08.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:06.984735+0000 mgr.smithi084.sihbqg (mgr.14162) 280 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:10.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:08.985047+0000 mgr.smithi084.sihbqg (mgr.14162) 281 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:10.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:08.985047+0000 mgr.smithi084.sihbqg (mgr.14162) 281 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:10.985400+0000 mgr.smithi084.sihbqg (mgr.14162) 282 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:10.985400+0000 mgr.smithi084.sihbqg (mgr.14162) 282 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:12.985863+0000 mgr.smithi084.sihbqg (mgr.14162) 283 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:12.985863+0000 mgr.smithi084.sihbqg (mgr.14162) 283 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:14.986169+0000 mgr.smithi084.sihbqg (mgr.14162) 284 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:14.986169+0000 mgr.smithi084.sihbqg (mgr.14162) 284 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:16.986438+0000 mgr.smithi084.sihbqg (mgr.14162) 285 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:18.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:16.986438+0000 mgr.smithi084.sihbqg (mgr.14162) 285 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:20.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:18.986915+0000 mgr.smithi084.sihbqg (mgr.14162) 286 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:20.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:18.986915+0000 mgr.smithi084.sihbqg (mgr.14162) 286 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:23.134 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:20.987373+0000 mgr.smithi084.sihbqg (mgr.14162) 287 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:23.134 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:22.173250+0000 mon.smithi084 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:28:23.245 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:20.987373+0000 mgr.smithi084.sihbqg (mgr.14162) 287 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:23.245 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:22.173250+0000 mon.smithi084 (mon.0) 348 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:28:24.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:28:22.172730+0000 mgr.smithi084.sihbqg (mgr.14162) 288 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:28:24.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:28:24.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:28:24.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:26:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:27:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:27:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:28:24.166 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:26:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.167 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:27:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:27:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:28:24.168 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:28:24.169 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:28:24.169 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:28:24.169 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:23.153765+0000 mon.smithi084 (mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-18T07:28:24.169 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:23.154344+0000 mon.smithi084 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:28:24.169 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:23.275644+0000 mon.smithi084 (mon.0) 351 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:24.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:28:22.172730+0000 mgr.smithi084.sihbqg (mgr.14162) 288 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:28:24.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:28:24.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:26:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:27:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:27:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:28:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:28:24.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:26:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:27:07Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:27:44Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:28:24.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:28:24.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:28:24.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:28:24.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:23.153765+0000 mon.smithi084 (mon.0) 349 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-18T07:28:24.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:23.154344+0000 mon.smithi084 (mon.0) 350 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:28:24.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:23.275644+0000 mon.smithi084 (mon.0) 351 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:25.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:22.987682+0000 mgr.smithi084.sihbqg (mgr.14162) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:25.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:28:23.154815+0000 mgr.smithi084.sihbqg (mgr.14162) 290 : cephadm [INF] Deploying daemon osd.3 on smithi106 2024-01-18T07:28:25.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:24.367299+0000 mon.smithi084 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:25.341 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:22.987682+0000 mgr.smithi084.sihbqg (mgr.14162) 289 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:25.341 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:28:23.154815+0000 mgr.smithi084.sihbqg (mgr.14162) 290 : cephadm [INF] Deploying daemon osd.3 on smithi106 2024-01-18T07:28:25.341 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:24.367299+0000 mon.smithi084 (mon.0) 352 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:25.577 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 3 on host 'smithi106' 2024-01-18T07:28:25.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.575+0000 7fe2717fa700 1 -- 172.21.15.106:0/3836215186 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe274057f10 con 0x7fe26405c2b0 2024-01-18T07:28:25.578 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 msgr2=0x7fe26405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 0x7fe26405e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fe260009fd0 tx=0x7fe26000b040).stop 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 msgr2=0x7fe274109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fe268000c00 tx=0x7fe26800b040).stop 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 shutdown_connections 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe26405c2b0 0x7fe26405e770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2740f55d0 0x7fe274109c30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 --2- 172.21.15.106:0/3836215186 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe2740f6870 0x7fe27410c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 >> 172.21.15.106:0/3836215186 conn(0x7fe2740f0d30 msgr2=0x7fe2740fa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 shutdown_connections 2024-01-18T07:28:25.579 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:25.577+0000 7fe27aea8700 1 -- 172.21.15.106:0/3836215186 wait complete. 2024-01-18T07:28:25.917 DEBUG:teuthology.orchestra.run.smithi106:osd.3> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.3.service 2024-01-18T07:28:25.920 INFO:tasks.cephadm:Deploying osd.4 on smithi106 with /dev/vg_nvme/lv_3... 2024-01-18T07:28:25.920 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-18T07:28:25.949 INFO:journalctl@ceph.osd.3.smithi106.stdout:-- Logs begin at Thu 2024-01-18 07:09:06 UTC. -- 2024-01-18T07:28:26.164 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:28:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:28:26.036+0000 7fcbe33ed080 -1 Falling back to public interface 2024-01-18T07:28:26.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:24.988117+0000 mgr.smithi084.sihbqg (mgr.14162) 291 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:26.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:25.572472+0000 mon.smithi084 (mon.0) 353 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:26.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:25.575846+0000 mon.smithi084 (mon.0) 354 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:26.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:25.577827+0000 mon.smithi084 (mon.0) 355 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-01-18T07:28:26.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:24.988117+0000 mgr.smithi084.sihbqg (mgr.14162) 291 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:26.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:25.572472+0000 mon.smithi084 (mon.0) 353 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:26.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:25.575846+0000 mon.smithi084 (mon.0) 354 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:26.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:25.577827+0000 mon.smithi084 (mon.0) 355 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2024-01-18T07:28:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-18T07:28:27.433 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-18T07:28:27.434 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-01-18T07:28:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-01-18T07:28:27.434 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125768 s, 834 MB/s 2024-01-18T07:28:27.434 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-01-18T07:28:27.765 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:26.579429+0000 mon.smithi084 (mon.0) 356 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-01-18T07:28:27.765 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:26.579474+0000 mon.smithi084 (mon.0) 357 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:26.579597+0000 mon.smithi084 (mon.0) 358 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:26.580400+0000 mon.smithi084 (mon.0) 359 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:28:26.686217+0000 mgr.smithi084.sihbqg (mgr.14162) 292 : cephadm [INF] Detected new or changed devices on smithi084 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:26.692682+0000 mon.smithi084 (mon.0) 360 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:26.933181+0000 mgr.smithi084.sihbqg (mgr.14162) 293 : cluster [DBG] pgmap v275: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:26.938497+0000 mon.smithi084 (mon.0) 361 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:27.766 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:28:26.942095+0000 mgr.smithi084.sihbqg (mgr.14162) 294 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:28:27.824 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:26.579429+0000 mon.smithi084 (mon.0) 356 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:26.579474+0000 mon.smithi084 (mon.0) 357 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:26.579597+0000 mon.smithi084 (mon.0) 358 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:26.580400+0000 mon.smithi084 (mon.0) 359 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:28:26.686217+0000 mgr.smithi084.sihbqg (mgr.14162) 292 : cephadm [INF] Detected new or changed devices on smithi084 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:26.692682+0000 mon.smithi084 (mon.0) 360 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:26.933181+0000 mgr.smithi084.sihbqg (mgr.14162) 293 : cluster [DBG] pgmap v275: 1 pgs: 1 unknown; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:27.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:26.938497+0000 mon.smithi084 (mon.0) 361 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:28:27.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:28:26.942095+0000 mgr.smithi084.sihbqg (mgr.14162) 294 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:28:28.764 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:27.581144+0000 mon.smithi084 (mon.0) 362 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-01-18T07:28:28.764 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:27.581235+0000 mon.smithi084 (mon.0) 363 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2024-01-18T07:28:28.764 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:27.581900+0000 mon.smithi084 (mon.0) 364 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:28.765 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:27.690985+0000 mon.smithi084 (mon.0) 365 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-01-18T07:28:28.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:27.581144+0000 mon.smithi084 (mon.0) 362 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-01-18T07:28:28.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:27.581235+0000 mon.smithi084 (mon.0) 363 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2024-01-18T07:28:28.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:27.581900+0000 mon.smithi084 (mon.0) 364 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:28.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:27.690985+0000 mon.smithi084 (mon.0) 365 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 -- 172.21.15.106:0/2060391994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 msgr2=0x7f511c0f7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 --2- 172.21.15.106:0/2060391994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c0f7de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5110004660 tx=0x7f5110009b30).stop 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 -- 172.21.15.106:0/2060391994 shutdown_connections 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 --2- 172.21.15.106:0/2060391994 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c0f7de0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 --2- 172.21.15.106:0/2060391994 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 0x7f511c0f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.954+0000 7f5120f57700 1 -- 172.21.15.106:0/2060391994 >> 172.21.15.106:0/2060391994 conn(0x7f511c0f1e40 msgr2=0x7f511c0f4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 -- 172.21.15.106:0/2060391994 shutdown_connections 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 -- 172.21.15.106:0/2060391994 wait complete. 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 Processor -- start 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 -- start start 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 0x7f511c106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:28:28.958 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c107b00 con 0x7f511c0f66e0 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5120f57700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1083b0 con 0x7f511c0f7980 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5119d9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5119d9b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:43862/0 (socket says 172.21.15.106:43862) 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f5119d9b700 1 -- 172.21.15.106:0/1924064727 learned_addr learned my addr 172.21.15.106:0/1924064727 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.955+0000 7f511a59c700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 0x7f511c106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5119d9b700 1 -- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 msgr2=0x7f511c106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5119d9b700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 0x7f511c106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5119d9b700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110005040 con 0x7f511c0f7980 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5119d9b700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5110007a40 tx=0x7f5110009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5110018070 con 0x7f511c0f7980 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5110013410 con 0x7f511c0f7980 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c108630 con 0x7f511c0f7980 2024-01-18T07:28:28.959 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f511001c620 con 0x7f511c0f7980 2024-01-18T07:28:28.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.956+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c108b90 con 0x7f511c0f7980 2024-01-18T07:28:28.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.957+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c0f5ad0 con 0x7f511c0f7980 2024-01-18T07:28:28.960 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.957+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f511001c8e0 con 0x7f511c0f7980 2024-01-18T07:28:28.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.957+0000 7f510affd700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 0x7f510405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:28:28.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.958+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3261+0+0 (secure 0 0 0) 0x7f51100775a0 con 0x7f511c0f7980 2024-01-18T07:28:28.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.958+0000 7f511a59c700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 0x7f510405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:28:28.961 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.958+0000 7f511a59c700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 0x7f510405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f511c0f77e0 tx=0x7f510c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:28:28.962 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:28.961+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f511004aa50 con 0x7f511c0f7980 2024-01-18T07:28:29.142 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:28:29.141+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f511c0fef80 con 0x7f510405c2a0 2024-01-18T07:28:29.702 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:28:29.529+0000 7fcbe33ed080 -1 osd.3 0 log_to_monitors {default=true} 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:28.699767+0000 mon.smithi084 (mon.0) 366 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:28.699930+0000 mon.smithi084 (mon.0) 367 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:28.933646+0000 mgr.smithi084.sihbqg (mgr.14162) 295 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.144922+0000 mon.smithi084 (mon.0) 368 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.146588+0000 mon.smithi084 (mon.0) 369 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:28:30.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.147362+0000 mon.smithi084 (mon.0) 370 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:28:30.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.533991+0000 mon.smithi106 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-18T07:28:30.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.534358+0000 mon.smithi084 (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:28.699767+0000 mon.smithi084 (mon.0) 366 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:28.699930+0000 mon.smithi084 (mon.0) 367 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:28.933646+0000 mgr.smithi084.sihbqg (mgr.14162) 295 : cluster [DBG] pgmap v278: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.144922+0000 mon.smithi084 (mon.0) 368 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.146588+0000 mon.smithi084 (mon.0) 369 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.147362+0000 mon.smithi084 (mon.0) 370 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:28:30.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.533991+0000 mon.smithi106 (mon.1) 4 : audit [INF] from='osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-18T07:28:30.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.534358+0000 mon.smithi084 (mon.0) 371 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.144044+0000 mgr.smithi084.sihbqg (mgr.14162) 296 : audit [DBG] from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:28:29.146273+0000 mgr.smithi084.sihbqg (mgr.14162) 297 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.703392+0000 mon.smithi084 (mon.0) 372 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:29.703461+0000 mon.smithi084 (mon.0) 373 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.703761+0000 mon.smithi084 (mon.0) 374 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:31.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.705280+0000 mon.smithi106 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:28:31.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:29.705633+0000 mon.smithi084 (mon.0) 375 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:28:31.164 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:28:30.708+0000 7fcbd995e700 -1 osd.3 0 waiting for initial osdmap 2024-01-18T07:28:31.164 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:28:30.715+0000 7fcbd30d4700 -1 osd.3 24 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.144044+0000 mgr.smithi084.sihbqg (mgr.14162) 296 : audit [DBG] from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:28:29.146273+0000 mgr.smithi084.sihbqg (mgr.14162) 297 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.703392+0000 mon.smithi084 (mon.0) 372 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:29.703461+0000 mon.smithi084 (mon.0) 373 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.703761+0000 mon.smithi084 (mon.0) 374 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.705280+0000 mon.smithi106 (mon.1) 5 : audit [INF] from='osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:28:31.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:29.705633+0000 mon.smithi084 (mon.0) 375 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:30.704955+0000 mon.smithi084 (mon.0) 376 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.705068+0000 mon.smithi084 (mon.0) 377 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:30.705503+0000 mon.smithi084 (mon.0) 378 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:30.709768+0000 mon.smithi084 (mon.0) 379 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.933950+0000 mgr.smithi084.sihbqg (mgr.14162) 298 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.962477+0000 mon.smithi084 (mon.0) 380 : cluster [INF] osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] boot 2024-01-18T07:28:32.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.962527+0000 mon.smithi084 (mon.0) 381 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-01-18T07:28:32.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:30.962888+0000 mon.smithi084 (mon.0) 382 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:30.704955+0000 mon.smithi084 (mon.0) 376 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.705068+0000 mon.smithi084 (mon.0) 377 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:30.705503+0000 mon.smithi084 (mon.0) 378 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:30.709768+0000 mon.smithi084 (mon.0) 379 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.933950+0000 mgr.smithi084.sihbqg (mgr.14162) 298 : cluster [DBG] pgmap v281: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.962477+0000 mon.smithi084 (mon.0) 380 : cluster [INF] osd.3 [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] boot 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.962527+0000 mon.smithi084 (mon.0) 381 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-01-18T07:28:32.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:30.962888+0000 mon.smithi084 (mon.0) 382 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-18T07:28:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.539157+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:28:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:30.539268+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:28:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:31.964269+0000 mon.smithi084 (mon.0) 383 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-01-18T07:28:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.539157+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:28:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:30.539268+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:28:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:31.964269+0000 mon.smithi084 (mon.0) 383 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-01-18T07:28:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:32.934291+0000 mgr.smithi084.sihbqg (mgr.14162) 299 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:32.968885+0000 mon.smithi084 (mon.0) 384 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-01-18T07:28:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:32.934291+0000 mgr.smithi084.sihbqg (mgr.14162) 299 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 0 B data, 19 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:32.968885+0000 mon.smithi084 (mon.0) 384 : cluster [DBG] osdmap e27: 4 total, 4 up, 4 in 2024-01-18T07:28:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:34.934591+0000 mgr.smithi084.sihbqg (mgr.14162) 300 : cluster [DBG] pgmap v286: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:34.934591+0000 mgr.smithi084.sihbqg (mgr.14162) 300 : cluster [DBG] pgmap v286: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:35.041422+0000 mon.smithi084 (mon.0) 385 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:28:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:28:35.050398+0000 mon.smithi084 (mon.0) 386 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:28:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:35.041422+0000 mon.smithi084 (mon.0) 385 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:28:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:28:35.050398+0000 mon.smithi084 (mon.0) 386 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:28:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:36.934947+0000 mgr.smithi084.sihbqg (mgr.14162) 301 : cluster [DBG] pgmap v287: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:36.934947+0000 mgr.smithi084.sihbqg (mgr.14162) 301 : cluster [DBG] pgmap v287: 1 pgs: 1 peering; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:38.935376+0000 mgr.smithi084.sihbqg (mgr.14162) 302 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:38.935376+0000 mgr.smithi084.sihbqg (mgr.14162) 302 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:40.935760+0000 mgr.smithi084.sihbqg (mgr.14162) 303 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:40.935760+0000 mgr.smithi084.sihbqg (mgr.14162) 303 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:42.936124+0000 mgr.smithi084.sihbqg (mgr.14162) 304 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:44.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:42.936124+0000 mgr.smithi084.sihbqg (mgr.14162) 304 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:44.936469+0000 mgr.smithi084.sihbqg (mgr.14162) 305 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:44.936469+0000 mgr.smithi084.sihbqg (mgr.14162) 305 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:48.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:46.936848+0000 mgr.smithi084.sihbqg (mgr.14162) 306 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:48.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:46.936848+0000 mgr.smithi084.sihbqg (mgr.14162) 306 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:48.937326+0000 mgr.smithi084.sihbqg (mgr.14162) 307 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:50.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:48.937326+0000 mgr.smithi084.sihbqg (mgr.14162) 307 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:52.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:50.937700+0000 mgr.smithi084.sihbqg (mgr.14162) 308 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:52.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:50.937700+0000 mgr.smithi084.sihbqg (mgr.14162) 308 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:54.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:52.938077+0000 mgr.smithi084.sihbqg (mgr.14162) 309 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:54.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:52.938077+0000 mgr.smithi084.sihbqg (mgr.14162) 309 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:54.938508+0000 mgr.smithi084.sihbqg (mgr.14162) 310 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:54.938508+0000 mgr.smithi084.sihbqg (mgr.14162) 310 : cluster [DBG] pgmap v296: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:56.938822+0000 mgr.smithi084.sihbqg (mgr.14162) 311 : cluster [DBG] pgmap v297: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:58.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:56.938822+0000 mgr.smithi084.sihbqg (mgr.14162) 311 : cluster [DBG] pgmap v297: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:28:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:28:58.939196+0000 mgr.smithi084.sihbqg (mgr.14162) 312 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:28:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:28:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:28:58.939196+0000 mgr.smithi084.sihbqg (mgr.14162) 312 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:00.939576+0000 mgr.smithi084.sihbqg (mgr.14162) 313 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:02.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:00.939576+0000 mgr.smithi084.sihbqg (mgr.14162) 313 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:04.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:02.939962+0000 mgr.smithi084.sihbqg (mgr.14162) 314 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:04.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:02.939962+0000 mgr.smithi084.sihbqg (mgr.14162) 314 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:04.940366+0000 mgr.smithi084.sihbqg (mgr.14162) 315 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:04.940366+0000 mgr.smithi084.sihbqg (mgr.14162) 315 : cluster [DBG] pgmap v301: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:08.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:06.940922+0000 mgr.smithi084.sihbqg (mgr.14162) 316 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:08.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:06.940922+0000 mgr.smithi084.sihbqg (mgr.14162) 316 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:08.941260+0000 mgr.smithi084.sihbqg (mgr.14162) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:08.941260+0000 mgr.smithi084.sihbqg (mgr.14162) 317 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:10.941718+0000 mgr.smithi084.sihbqg (mgr.14162) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:12.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:10.941718+0000 mgr.smithi084.sihbqg (mgr.14162) 318 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:12.942117+0000 mgr.smithi084.sihbqg (mgr.14162) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:12.942117+0000 mgr.smithi084.sihbqg (mgr.14162) 319 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:14.942421+0000 mgr.smithi084.sihbqg (mgr.14162) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:14.942421+0000 mgr.smithi084.sihbqg (mgr.14162) 320 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:16.942730+0000 mgr.smithi084.sihbqg (mgr.14162) 321 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:18.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:16.942730+0000 mgr.smithi084.sihbqg (mgr.14162) 321 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:18.943030+0000 mgr.smithi084.sihbqg (mgr.14162) 322 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:18.943030+0000 mgr.smithi084.sihbqg (mgr.14162) 322 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:22.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:20.943407+0000 mgr.smithi084.sihbqg (mgr.14162) 323 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:22.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:20.943407+0000 mgr.smithi084.sihbqg (mgr.14162) 323 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:22.943799+0000 mgr.smithi084.sihbqg (mgr.14162) 324 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:22.943799+0000 mgr.smithi084.sihbqg (mgr.14162) 324 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:24.944179+0000 mgr.smithi084.sihbqg (mgr.14162) 325 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:24.944179+0000 mgr.smithi084.sihbqg (mgr.14162) 325 : cluster [DBG] pgmap v311: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:26.944492+0000 mgr.smithi084.sihbqg (mgr.14162) 326 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:26.944492+0000 mgr.smithi084.sihbqg (mgr.14162) 326 : cluster [DBG] pgmap v312: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:28.944841+0000 mgr.smithi084.sihbqg (mgr.14162) 327 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:28.944841+0000 mgr.smithi084.sihbqg (mgr.14162) 327 : cluster [DBG] pgmap v313: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:32.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:30.945159+0000 mgr.smithi084.sihbqg (mgr.14162) 328 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:32.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:30.945159+0000 mgr.smithi084.sihbqg (mgr.14162) 328 : cluster [DBG] pgmap v314: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:32.945534+0000 mgr.smithi084.sihbqg (mgr.14162) 329 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:32.945534+0000 mgr.smithi084.sihbqg (mgr.14162) 329 : cluster [DBG] pgmap v315: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:34.946007+0000 mgr.smithi084.sihbqg (mgr.14162) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:34.946007+0000 mgr.smithi084.sihbqg (mgr.14162) 330 : cluster [DBG] pgmap v316: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:29:35.042948+0000 mon.smithi084 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:29:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:29:35.051544+0000 mon.smithi084 (mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:29:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:29:35.042948+0000 mon.smithi084 (mon.0) 387 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:29:36.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:29:35.051544+0000 mon.smithi084 (mon.0) 388 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:29:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:36.946451+0000 mgr.smithi084.sihbqg (mgr.14162) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:36.946451+0000 mgr.smithi084.sihbqg (mgr.14162) 331 : cluster [DBG] pgmap v317: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:38.946926+0000 mgr.smithi084.sihbqg (mgr.14162) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:38.946926+0000 mgr.smithi084.sihbqg (mgr.14162) 332 : cluster [DBG] pgmap v318: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:40.947373+0000 mgr.smithi084.sihbqg (mgr.14162) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:40.947373+0000 mgr.smithi084.sihbqg (mgr.14162) 333 : cluster [DBG] pgmap v319: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:42.947892+0000 mgr.smithi084.sihbqg (mgr.14162) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:42.947892+0000 mgr.smithi084.sihbqg (mgr.14162) 334 : cluster [DBG] pgmap v320: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:44.948357+0000 mgr.smithi084.sihbqg (mgr.14162) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:44.948357+0000 mgr.smithi084.sihbqg (mgr.14162) 335 : cluster [DBG] pgmap v321: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:46.948869+0000 mgr.smithi084.sihbqg (mgr.14162) 336 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:46.948869+0000 mgr.smithi084.sihbqg (mgr.14162) 336 : cluster [DBG] pgmap v322: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:48.949390+0000 mgr.smithi084.sihbqg (mgr.14162) 337 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:48.949390+0000 mgr.smithi084.sihbqg (mgr.14162) 337 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:52.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:50.949896+0000 mgr.smithi084.sihbqg (mgr.14162) 338 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:52.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:50.949896+0000 mgr.smithi084.sihbqg (mgr.14162) 338 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:52.950228+0000 mgr.smithi084.sihbqg (mgr.14162) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:52.950228+0000 mgr.smithi084.sihbqg (mgr.14162) 339 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:54.950669+0000 mgr.smithi084.sihbqg (mgr.14162) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:54.950669+0000 mgr.smithi084.sihbqg (mgr.14162) 340 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:56.951039+0000 mgr.smithi084.sihbqg (mgr.14162) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:56.951039+0000 mgr.smithi084.sihbqg (mgr.14162) 341 : cluster [DBG] pgmap v327: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:29:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:29:58.951427+0000 mgr.smithi084.sihbqg (mgr.14162) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:29:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:29:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:29:58.951427+0000 mgr.smithi084.sihbqg (mgr.14162) 342 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:00.000162+0000 mon.smithi084 (mon.0) 389 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-01-18T07:30:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:00.000198+0000 mon.smithi084 (mon.0) 390 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-01-18T07:30:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:00.000226+0000 mon.smithi084 (mon.0) 391 : cluster [WRN] daemon node-exporter.smithi106 on smithi106 is in unknown state 2024-01-18T07:30:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:00.000162+0000 mon.smithi084 (mon.0) 389 : cluster [WRN] Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-01-18T07:30:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:00.000198+0000 mon.smithi084 (mon.0) 390 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-01-18T07:30:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:00.000226+0000 mon.smithi084 (mon.0) 391 : cluster [WRN] daemon node-exporter.smithi106 on smithi106 is in unknown state 2024-01-18T07:30:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:00.951893+0000 mgr.smithi084.sihbqg (mgr.14162) 343 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:00.951893+0000 mgr.smithi084.sihbqg (mgr.14162) 343 : cluster [DBG] pgmap v329: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:02.952261+0000 mgr.smithi084.sihbqg (mgr.14162) 344 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:02.952261+0000 mgr.smithi084.sihbqg (mgr.14162) 344 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:05.282 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:04.952590+0000 mgr.smithi084.sihbqg (mgr.14162) 345 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:04.952590+0000 mgr.smithi084.sihbqg (mgr.14162) 345 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:06.952955+0000 mgr.smithi084.sihbqg (mgr.14162) 346 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:06.952955+0000 mgr.smithi084.sihbqg (mgr.14162) 346 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:08.953280+0000 mgr.smithi084.sihbqg (mgr.14162) 347 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:08.953280+0000 mgr.smithi084.sihbqg (mgr.14162) 347 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:10.953744+0000 mgr.smithi084.sihbqg (mgr.14162) 348 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:10.953744+0000 mgr.smithi084.sihbqg (mgr.14162) 348 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:12.954253+0000 mgr.smithi084.sihbqg (mgr.14162) 349 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:12.954253+0000 mgr.smithi084.sihbqg (mgr.14162) 349 : cluster [DBG] pgmap v335: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:14.954731+0000 mgr.smithi084.sihbqg (mgr.14162) 350 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:14.954731+0000 mgr.smithi084.sihbqg (mgr.14162) 350 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:16.955071+0000 mgr.smithi084.sihbqg (mgr.14162) 351 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:16.955071+0000 mgr.smithi084.sihbqg (mgr.14162) 351 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:18.955405+0000 mgr.smithi084.sihbqg (mgr.14162) 352 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:18.955405+0000 mgr.smithi084.sihbqg (mgr.14162) 352 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:20.955899+0000 mgr.smithi084.sihbqg (mgr.14162) 353 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:20.955899+0000 mgr.smithi084.sihbqg (mgr.14162) 353 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:22.956294+0000 mgr.smithi084.sihbqg (mgr.14162) 354 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:22.956294+0000 mgr.smithi084.sihbqg (mgr.14162) 354 : cluster [DBG] pgmap v340: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:24.956727+0000 mgr.smithi084.sihbqg (mgr.14162) 355 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:24.956727+0000 mgr.smithi084.sihbqg (mgr.14162) 355 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:26.957186+0000 mgr.smithi084.sihbqg (mgr.14162) 356 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:26.957186+0000 mgr.smithi084.sihbqg (mgr.14162) 356 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:28.957676+0000 mgr.smithi084.sihbqg (mgr.14162) 357 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:28.957676+0000 mgr.smithi084.sihbqg (mgr.14162) 357 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:30.958009+0000 mgr.smithi084.sihbqg (mgr.14162) 358 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:30.958009+0000 mgr.smithi084.sihbqg (mgr.14162) 358 : cluster [DBG] pgmap v344: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:32.958349+0000 mgr.smithi084.sihbqg (mgr.14162) 359 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:32.958349+0000 mgr.smithi084.sihbqg (mgr.14162) 359 : cluster [DBG] pgmap v345: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:34.958733+0000 mgr.smithi084.sihbqg (mgr.14162) 360 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:34.958733+0000 mgr.smithi084.sihbqg (mgr.14162) 360 : cluster [DBG] pgmap v346: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:36.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:30:35.044528+0000 mon.smithi084 (mon.0) 392 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:30:36.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:30:35.052713+0000 mon.smithi084 (mon.0) 393 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:30:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:30:35.044528+0000 mon.smithi084 (mon.0) 392 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:30:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:30:35.052713+0000 mon.smithi084 (mon.0) 393 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:30:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:36.959024+0000 mgr.smithi084.sihbqg (mgr.14162) 361 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:36.959024+0000 mgr.smithi084.sihbqg (mgr.14162) 361 : cluster [DBG] pgmap v347: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:38.959379+0000 mgr.smithi084.sihbqg (mgr.14162) 362 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:38.959379+0000 mgr.smithi084.sihbqg (mgr.14162) 362 : cluster [DBG] pgmap v348: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:40.959706+0000 mgr.smithi084.sihbqg (mgr.14162) 363 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:40.959706+0000 mgr.smithi084.sihbqg (mgr.14162) 363 : cluster [DBG] pgmap v349: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:42.960173+0000 mgr.smithi084.sihbqg (mgr.14162) 364 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:42.960173+0000 mgr.smithi084.sihbqg (mgr.14162) 364 : cluster [DBG] pgmap v350: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:44.960698+0000 mgr.smithi084.sihbqg (mgr.14162) 365 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:44.960698+0000 mgr.smithi084.sihbqg (mgr.14162) 365 : cluster [DBG] pgmap v351: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:46.961200+0000 mgr.smithi084.sihbqg (mgr.14162) 366 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:46.961200+0000 mgr.smithi084.sihbqg (mgr.14162) 366 : cluster [DBG] pgmap v352: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:48.961569+0000 mgr.smithi084.sihbqg (mgr.14162) 367 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:48.961569+0000 mgr.smithi084.sihbqg (mgr.14162) 367 : cluster [DBG] pgmap v353: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:50.962003+0000 mgr.smithi084.sihbqg (mgr.14162) 368 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:50.962003+0000 mgr.smithi084.sihbqg (mgr.14162) 368 : cluster [DBG] pgmap v354: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:52.962483+0000 mgr.smithi084.sihbqg (mgr.14162) 369 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:52.962483+0000 mgr.smithi084.sihbqg (mgr.14162) 369 : cluster [DBG] pgmap v355: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:54.962961+0000 mgr.smithi084.sihbqg (mgr.14162) 370 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:54.962961+0000 mgr.smithi084.sihbqg (mgr.14162) 370 : cluster [DBG] pgmap v356: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:56.963256+0000 mgr.smithi084.sihbqg (mgr.14162) 371 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:56.963256+0000 mgr.smithi084.sihbqg (mgr.14162) 371 : cluster [DBG] pgmap v357: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:30:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:30:58.963734+0000 mgr.smithi084.sihbqg (mgr.14162) 372 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:30:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:30:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:30:58.963734+0000 mgr.smithi084.sihbqg (mgr.14162) 372 : cluster [DBG] pgmap v358: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:00.964201+0000 mgr.smithi084.sihbqg (mgr.14162) 373 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:00.964201+0000 mgr.smithi084.sihbqg (mgr.14162) 373 : cluster [DBG] pgmap v359: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:02.964557+0000 mgr.smithi084.sihbqg (mgr.14162) 374 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:02.964557+0000 mgr.smithi084.sihbqg (mgr.14162) 374 : cluster [DBG] pgmap v360: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:04.965043+0000 mgr.smithi084.sihbqg (mgr.14162) 375 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:04.965043+0000 mgr.smithi084.sihbqg (mgr.14162) 375 : cluster [DBG] pgmap v361: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:06.965515+0000 mgr.smithi084.sihbqg (mgr.14162) 376 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:06.965515+0000 mgr.smithi084.sihbqg (mgr.14162) 376 : cluster [DBG] pgmap v362: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:08.965889+0000 mgr.smithi084.sihbqg (mgr.14162) 377 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:08.965889+0000 mgr.smithi084.sihbqg (mgr.14162) 377 : cluster [DBG] pgmap v363: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:10.966236+0000 mgr.smithi084.sihbqg (mgr.14162) 378 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:10.966236+0000 mgr.smithi084.sihbqg (mgr.14162) 378 : cluster [DBG] pgmap v364: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:12.966712+0000 mgr.smithi084.sihbqg (mgr.14162) 379 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:12.966712+0000 mgr.smithi084.sihbqg (mgr.14162) 379 : cluster [DBG] pgmap v365: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:14.967164+0000 mgr.smithi084.sihbqg (mgr.14162) 380 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:14.967164+0000 mgr.smithi084.sihbqg (mgr.14162) 380 : cluster [DBG] pgmap v366: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:16.967479+0000 mgr.smithi084.sihbqg (mgr.14162) 381 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:16.967479+0000 mgr.smithi084.sihbqg (mgr.14162) 381 : cluster [DBG] pgmap v367: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:18.967787+0000 mgr.smithi084.sihbqg (mgr.14162) 382 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:18.967787+0000 mgr.smithi084.sihbqg (mgr.14162) 382 : cluster [DBG] pgmap v368: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:20.968144+0000 mgr.smithi084.sihbqg (mgr.14162) 383 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:20.968144+0000 mgr.smithi084.sihbqg (mgr.14162) 383 : cluster [DBG] pgmap v369: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:22.968486+0000 mgr.smithi084.sihbqg (mgr.14162) 384 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:22.968486+0000 mgr.smithi084.sihbqg (mgr.14162) 384 : cluster [DBG] pgmap v370: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:24.968878+0000 mgr.smithi084.sihbqg (mgr.14162) 385 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:24.968878+0000 mgr.smithi084.sihbqg (mgr.14162) 385 : cluster [DBG] pgmap v371: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:26.467151+0000 mgr.smithi084.sihbqg (mgr.14162) 386 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:29:03Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:29:41Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:30:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:31:27.134 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:29:03Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:29:41Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:30:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:31:27.135 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:26.467518+0000 mgr.smithi084.sihbqg (mgr.14162) 387 : cephadm [INF] Reconfiguring mon.smithi084 (unknown last config time)... 2024-01-18T07:31:27.136 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:26.467860+0000 mon.smithi084 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-18T07:31:27.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:26.468644+0000 mon.smithi084 (mon.0) 395 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-18T07:31:27.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:26.469352+0000 mon.smithi084 (mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:27.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:26.469937+0000 mgr.smithi084.sihbqg (mgr.14162) 388 : cephadm [INF] Reconfiguring daemon mon.smithi084 on smithi084 2024-01-18T07:31:27.137 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:26.969104+0000 mgr.smithi084.sihbqg (mgr.14162) 389 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:27.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:26.467151+0000 mgr.smithi084.sihbqg (mgr.14162) 386 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:29:03Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:29:41Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:30:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:31:27.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:31:27.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:29:03Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:29:41Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:30:48Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:31:27.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:26.467518+0000 mgr.smithi084.sihbqg (mgr.14162) 387 : cephadm [INF] Reconfiguring mon.smithi084 (unknown last config time)... 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:26.467860+0000 mon.smithi084 (mon.0) 394 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-18T07:31:27.419 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:26.468644+0000 mon.smithi084 (mon.0) 395 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-18T07:31:27.420 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:26.469352+0000 mon.smithi084 (mon.0) 396 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:27.420 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:26.469937+0000 mgr.smithi084.sihbqg (mgr.14162) 388 : cephadm [INF] Reconfiguring daemon mon.smithi084 on smithi084 2024-01-18T07:31:27.420 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:26.969104+0000 mgr.smithi084.sihbqg (mgr.14162) 389 : cluster [DBG] pgmap v372: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.303815+0000 mon.smithi106 (mon.1) 6 : audit [INF] from='client.? 172.21.15.106:0/2623296291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]: dispatch 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.304489+0000 mon.smithi084 (mon.0) 397 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]: dispatch 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.308162+0000 mon.smithi084 (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]': finished 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:27.308206+0000 mon.smithi084 (mon.0) 399 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.308310+0000 mon.smithi084 (mon.0) 400 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.386585+0000 mon.smithi084 (mon.0) 401 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:28.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:27.387083+0000 mgr.smithi084.sihbqg (mgr.14162) 390 : cephadm [INF] Reconfiguring mgr.smithi084.sihbqg (unknown last config time)... 2024-01-18T07:31:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.387295+0000 mon.smithi084 (mon.0) 402 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi084.sihbqg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-18T07:31:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.387861+0000 mon.smithi084 (mon.0) 403 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-18T07:31:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.388317+0000 mon.smithi084 (mon.0) 404 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:27.388703+0000 mgr.smithi084.sihbqg (mgr.14162) 391 : cephadm [INF] Reconfiguring daemon mgr.smithi084.sihbqg on smithi084 2024-01-18T07:31:28.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:27.716018+0000 mon.smithi106 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.106:0/4203828578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:31:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.303815+0000 mon.smithi106 (mon.1) 6 : audit [INF] from='client.? 172.21.15.106:0/2623296291' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]: dispatch 2024-01-18T07:31:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.304489+0000 mon.smithi084 (mon.0) 397 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]: dispatch 2024-01-18T07:31:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.308162+0000 mon.smithi084 (mon.0) 398 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "def49df5-fdbc-421f-9787-8017f3349879"}]': finished 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:27.308206+0000 mon.smithi084 (mon.0) 399 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.308310+0000 mon.smithi084 (mon.0) 400 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.386585+0000 mon.smithi084 (mon.0) 401 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:27.387083+0000 mgr.smithi084.sihbqg (mgr.14162) 390 : cephadm [INF] Reconfiguring mgr.smithi084.sihbqg (unknown last config time)... 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.387295+0000 mon.smithi084 (mon.0) 402 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi084.sihbqg", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.387861+0000 mon.smithi084 (mon.0) 403 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.388317+0000 mon.smithi084 (mon.0) 404 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:27.388703+0000 mgr.smithi084.sihbqg (mgr.14162) 391 : cephadm [INF] Reconfiguring daemon mgr.smithi084.sihbqg on smithi084 2024-01-18T07:31:28.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:27.716018+0000 mon.smithi106 (mon.1) 7 : audit [DBG] from='client.? 172.21.15.106:0/4203828578' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:31:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:28.978+0000 7f5109ffb700 1 -- 172.21.15.106:0/1924064727 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50f80031e0 con 0x7f511c0f7980 2024-01-18T07:31:28.979 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:28.979+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mon.1 v2:172.21.15.106:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f50f80031e0 con 0x7f511c0f7980 2024-01-18T07:31:29.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:28.251573+0000 mon.smithi084 (mon.0) 405 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:29.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:28.252161+0000 mgr.smithi084.sihbqg (mgr.14162) 392 : cephadm [INF] Reconfiguring alertmanager.smithi084 (dependencies changed)... 2024-01-18T07:31:29.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:28.254169+0000 mgr.smithi084.sihbqg (mgr.14162) 393 : cephadm [INF] Reconfiguring daemon alertmanager.smithi084 on smithi084 2024-01-18T07:31:29.540 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:28.969472+0000 mgr.smithi084.sihbqg (mgr.14162) 394 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:29.600 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:28.251573+0000 mon.smithi084 (mon.0) 405 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:29.600 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:28.252161+0000 mgr.smithi084.sihbqg (mgr.14162) 392 : cephadm [INF] Reconfiguring alertmanager.smithi084 (dependencies changed)... 2024-01-18T07:31:29.600 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:28.254169+0000 mgr.smithi084.sihbqg (mgr.14162) 393 : cephadm [INF] Reconfiguring daemon alertmanager.smithi084 on smithi084 2024-01-18T07:31:29.600 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:28.969472+0000 mgr.smithi084.sihbqg (mgr.14162) 394 : cluster [DBG] pgmap v374: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:30.593205+0000 mon.smithi084 (mon.0) 406 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:30.593713+0000 mgr.smithi084.sihbqg (mgr.14162) 395 : cephadm [INF] Reconfiguring crash.smithi084 (monmap changed)... 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:30.593915+0000 mon.smithi084 (mon.0) 407 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:30.594459+0000 mon.smithi084 (mon.0) 408 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:30.594853+0000 mgr.smithi084.sihbqg (mgr.14162) 396 : cephadm [INF] Reconfiguring daemon crash.smithi084 on smithi084 2024-01-18T07:31:31.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:30.969916+0000 mgr.smithi084.sihbqg (mgr.14162) 397 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:31.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:31.494289+0000 mon.smithi084 (mon.0) 409 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:31.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:31.495237+0000 mon.smithi084 (mon.0) 410 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi106", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:31:31.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:31.495786+0000 mon.smithi084 (mon.0) 411 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:31.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:30.593205+0000 mon.smithi084 (mon.0) 406 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:31.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:30.593713+0000 mgr.smithi084.sihbqg (mgr.14162) 395 : cephadm [INF] Reconfiguring crash.smithi084 (monmap changed)... 2024-01-18T07:31:31.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:30.593915+0000 mon.smithi084 (mon.0) 407 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi084", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:31:31.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:30.594459+0000 mon.smithi084 (mon.0) 408 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:31.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:30.594853+0000 mgr.smithi084.sihbqg (mgr.14162) 396 : cephadm [INF] Reconfiguring daemon crash.smithi084 on smithi084 2024-01-18T07:31:31.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:30.969916+0000 mgr.smithi084.sihbqg (mgr.14162) 397 : cluster [DBG] pgmap v375: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:31.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:31.494289+0000 mon.smithi084 (mon.0) 409 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:31.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:31.495237+0000 mon.smithi084 (mon.0) 410 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi106", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-18T07:31:31.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:31.495786+0000 mon.smithi084 (mon.0) 411 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:32.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:31.494961+0000 mgr.smithi084.sihbqg (mgr.14162) 398 : cephadm [INF] Reconfiguring crash.smithi106 (monmap changed)... 2024-01-18T07:31:32.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:31.496154+0000 mgr.smithi084.sihbqg (mgr.14162) 399 : cephadm [INF] Reconfiguring daemon crash.smithi106 on smithi106 2024-01-18T07:31:32.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:32.575783+0000 mon.smithi084 (mon.0) 412 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:32.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:32.576621+0000 mon.smithi084 (mon.0) 413 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi106.yhaxbc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-18T07:31:32.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:32.577401+0000 mon.smithi084 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-18T07:31:32.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:32.578066+0000 mon.smithi084 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:31.494961+0000 mgr.smithi084.sihbqg (mgr.14162) 398 : cephadm [INF] Reconfiguring crash.smithi106 (monmap changed)... 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:31.496154+0000 mgr.smithi084.sihbqg (mgr.14162) 399 : cephadm [INF] Reconfiguring daemon crash.smithi106 on smithi106 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:32.575783+0000 mon.smithi084 (mon.0) 412 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:32.576621+0000 mon.smithi084 (mon.0) 413 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi106.yhaxbc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:32.577401+0000 mon.smithi084 (mon.0) 414 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-18T07:31:32.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:32.578066+0000 mon.smithi084 (mon.0) 415 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:33.646 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:32.576335+0000 mgr.smithi084.sihbqg (mgr.14162) 400 : cephadm [INF] Reconfiguring mgr.smithi106.yhaxbc (monmap changed)... 2024-01-18T07:31:33.646 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:32.578627+0000 mgr.smithi084.sihbqg (mgr.14162) 401 : cephadm [INF] Reconfiguring daemon mgr.smithi106.yhaxbc on smithi106 2024-01-18T07:31:33.646 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:32.970391+0000 mgr.smithi084.sihbqg (mgr.14162) 402 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:33.646 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:33.460142+0000 mon.smithi084 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-18T07:31:33.646 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:33.460930+0000 mon.smithi084 (mon.0) 417 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:33.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:32.576335+0000 mgr.smithi084.sihbqg (mgr.14162) 400 : cephadm [INF] Reconfiguring mgr.smithi106.yhaxbc (monmap changed)... 2024-01-18T07:31:33.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:32.578627+0000 mgr.smithi084.sihbqg (mgr.14162) 401 : cephadm [INF] Reconfiguring daemon mgr.smithi106.yhaxbc on smithi106 2024-01-18T07:31:33.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:32.970391+0000 mgr.smithi084.sihbqg (mgr.14162) 402 : cluster [DBG] pgmap v376: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:33.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:33.460142+0000 mon.smithi084 (mon.0) 416 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-18T07:31:33.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:33.460930+0000 mon.smithi084 (mon.0) 417 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:34.663 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:33.461558+0000 mgr.smithi084.sihbqg (mgr.14162) 403 : cephadm [INF] Deploying daemon osd.4 on smithi106 2024-01-18T07:31:34.663 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:34.181034+0000 mon.smithi084 (mon.0) 418 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:34.663 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:34.181910+0000 mon.smithi084 (mon.0) 419 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-18T07:31:34.663 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:34.182593+0000 mon.smithi084 (mon.0) 420 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-18T07:31:34.663 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:34.183232+0000 mon.smithi084 (mon.0) 421 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:34.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:33.461558+0000 mgr.smithi084.sihbqg (mgr.14162) 403 : cephadm [INF] Deploying daemon osd.4 on smithi106 2024-01-18T07:31:34.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:34.181034+0000 mon.smithi084 (mon.0) 418 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:34.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:34.181910+0000 mon.smithi084 (mon.0) 419 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-18T07:31:34.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:34.182593+0000 mon.smithi084 (mon.0) 420 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-18T07:31:34.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:34.183232+0000 mon.smithi084 (mon.0) 421 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:35.871 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:34.181601+0000 mgr.smithi084.sihbqg (mgr.14162) 404 : cephadm [INF] Reconfiguring mon.smithi106 (monmap changed)... 2024-01-18T07:31:35.872 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:34.183806+0000 mgr.smithi084.sihbqg (mgr.14162) 405 : cephadm [INF] Reconfiguring daemon mon.smithi106 on smithi106 2024-01-18T07:31:35.872 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:34.970749+0000 mgr.smithi084.sihbqg (mgr.14162) 406 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:35.872 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:35.045759+0000 mon.smithi084 (mon.0) 422 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:31:35.872 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:35.054355+0000 mon.smithi084 (mon.0) 423 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:31:35.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:34.181601+0000 mgr.smithi084.sihbqg (mgr.14162) 404 : cephadm [INF] Reconfiguring mon.smithi106 (monmap changed)... 2024-01-18T07:31:35.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:34.183806+0000 mgr.smithi084.sihbqg (mgr.14162) 405 : cephadm [INF] Reconfiguring daemon mon.smithi106 on smithi106 2024-01-18T07:31:35.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:34.970749+0000 mgr.smithi084.sihbqg (mgr.14162) 406 : cluster [DBG] pgmap v377: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:35.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:35.045759+0000 mon.smithi084 (mon.0) 422 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:31:35.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:35.054355+0000 mon.smithi084 (mon.0) 423 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:31:36.393 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 4 on host 'smithi106' 2024-01-18T07:31:36.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.392+0000 7f510affd700 1 -- 172.21.15.106:0/1924064727 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f511c0fef80 con 0x7f510405c2a0 2024-01-18T07:31:36.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 msgr2=0x7f510405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:31:36.394 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 0x7f510405e760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f511c0f77e0 tx=0x7f510c008040).stop 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 msgr2=0x7f511c1074c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5110007a40 tx=0x7f5110009b30).stop 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 shutdown_connections 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f511c0f66e0 0x7f511c106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f510405c2a0 0x7f510405e760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 --2- 172.21.15.106:0/1924064727 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f511c0f7980 0x7f511c1074c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 >> 172.21.15.106:0/1924064727 conn(0x7f511c0f1e40 msgr2=0x7f511c0fb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 shutdown_connections 2024-01-18T07:31:36.395 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:36.394+0000 7f5120f57700 1 -- 172.21.15.106:0/1924064727 wait complete. 2024-01-18T07:31:36.978 DEBUG:teuthology.orchestra.run.smithi106:osd.4> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.4.service 2024-01-18T07:31:36.980 INFO:tasks.cephadm:Deploying osd.5 on smithi106 with /dev/vg_nvme/lv_2... 2024-01-18T07:31:36.981 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-18T07:31:37.009 INFO:journalctl@ceph.osd.4.smithi106.stdout:-- Logs begin at Thu 2024-01-18 07:09:06 UTC. -- 2024-01-18T07:31:37.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:36.387921+0000 mon.smithi084 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:37.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:36.391719+0000 mon.smithi084 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:37.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:36.971169+0000 mgr.smithi084.sihbqg (mgr.14162) 407 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:37.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:37.252421+0000 mon.smithi084 (mon.0) 426 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:37.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:36.387921+0000 mon.smithi084 (mon.0) 424 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:37.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:36.391719+0000 mon.smithi084 (mon.0) 425 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:37.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:36.971169+0000 mgr.smithi084.sihbqg (mgr.14162) 407 : cluster [DBG] pgmap v378: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:37.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:37.252421+0000 mon.smithi084 (mon.0) 426 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:31:38.434 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-18T07:31:38.530 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-18T07:31:38.530 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-01-18T07:31:38.530 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-01-18T07:31:38.530 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120282 s, 872 MB/s 2024-01-18T07:31:38.530 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-01-18T07:31:38.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:37.253014+0000 mgr.smithi084.sihbqg (mgr.14162) 408 : cephadm [INF] Reconfiguring node-exporter.smithi106 (unknown last config time)... 2024-01-18T07:31:38.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:37.253155+0000 mgr.smithi084.sihbqg (mgr.14162) 409 : cephadm [INF] Reconfiguring daemon node-exporter.smithi106 on smithi106 2024-01-18T07:31:38.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-01-18T07:31:38.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:37.253014+0000 mgr.smithi084.sihbqg (mgr.14162) 408 : cephadm [INF] Reconfiguring node-exporter.smithi106 (unknown last config time)... 2024-01-18T07:31:38.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:37.253155+0000 mgr.smithi084.sihbqg (mgr.14162) 409 : cephadm [INF] Reconfiguring daemon node-exporter.smithi106 on smithi106 2024-01-18T07:31:39.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:38.971548+0000 mgr.smithi084.sihbqg (mgr.14162) 410 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:39.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:38.971548+0000 mgr.smithi084.sihbqg (mgr.14162) 410 : cluster [DBG] pgmap v379: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:39.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 -- 172.21.15.106:0/380481778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 msgr2=0x7ff5a80f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:31:39.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 --2- 172.21.15.106:0/380481778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a80f6cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7ff5a4004660 tx=0x7ff5a4009b30).stop 2024-01-18T07:31:39.923 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 -- 172.21.15.106:0/380481778 shutdown_connections 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 --2- 172.21.15.106:0/380481778 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a80f6cc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 --2- 172.21.15.106:0/380481778 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 0x7ff5a80f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 -- 172.21.15.106:0/380481778 >> 172.21.15.106:0/380481778 conn(0x7ff5a80f0d40 msgr2=0x7ff5a80f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 -- 172.21.15.106:0/380481778 shutdown_connections 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.920+0000 7ff5af542700 1 -- 172.21.15.106:0/380481778 wait complete. 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 Processor -- start 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 -- start start 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 0x7ff5a8105e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8106920 con 0x7ff5a80f55c0 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5af542700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8107130 con 0x7ff5a80f6860 2024-01-18T07:31:39.924 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:47776/0 (socket says 172.21.15.106:47776) 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 -- 172.21.15.106:0/2026834717 learned_addr learned my addr 172.21.15.106:0/2026834717 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5ad2de700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 0x7ff5a8105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 -- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 msgr2=0x7ff5a8105e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 0x7ff5a8105e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 -- 172.21.15.106:0/2026834717 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a4005040 con 0x7ff5a80f6860 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.921+0000 7ff5acadd700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff5a4000c00 tx=0x7ff5a400b040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.922+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a4019070 con 0x7ff5a80f6860 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.922+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a8107350 con 0x7ff5a80f6860 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.922+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5a4007e70 con 0x7ff5a80f6860 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.922+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff5a4014c90 con 0x7ff5a80f6860 2024-01-18T07:31:39.925 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.922+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a81078b0 con 0x7ff5a80f6860 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.923+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a80f49b0 con 0x7ff5a80f6860 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.923+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff5a4025070 con 0x7ff5a80f6860 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.923+0000 7ff59dffb700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 0x7ff59405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.924+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3803+0+0 (secure 0 0 0) 0x7ff5a40768f0 con 0x7ff5a80f6860 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.924+0000 7ff5ad2de700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 0x7ff59405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:31:39.926 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.924+0000 7ff5ad2de700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 0x7ff59405e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff5a8105ba0 tx=0x7ff59800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:31:39.928 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:39.927+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5a4049be0 con 0x7ff5a80f6860 2024-01-18T07:31:40.106 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:31:40.105+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5a80fdee0 con 0x7ff59405c2b0 2024-01-18T07:31:40.664 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:31:40.369+0000 7fc0556ac080 -1 osd.4 0 log_to_monitors {default=true} 2024-01-18T07:31:40.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.109221+0000 mon.smithi084 (mon.0) 427 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:31:40.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.110982+0000 mon.smithi084 (mon.0) 428 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:31:40.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.111659+0000 mon.smithi084 (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:40.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.373566+0000 mon.smithi106 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-18T07:31:40.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.373963+0000 mon.smithi084 (mon.0) 430 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-18T07:31:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.109221+0000 mon.smithi084 (mon.0) 427 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-18T07:31:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.110982+0000 mon.smithi084 (mon.0) 428 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-18T07:31:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.111659+0000 mon.smithi084 (mon.0) 429 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:31:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.373566+0000 mon.smithi106 (mon.1) 8 : audit [INF] from='osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-18T07:31:40.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.373963+0000 mon.smithi084 (mon.0) 430 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-18T07:31:41.664 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:31:41.408+0000 7fc04bc1d700 -1 osd.4 0 waiting for initial osdmap 2024-01-18T07:31:41.665 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:31:41.415+0000 7fc047397700 -1 osd.4 30 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.108329+0000 mgr.smithi084.sihbqg (mgr.14162) 411 : audit [DBG] from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:31:40.110708+0000 mgr.smithi084.sihbqg (mgr.14162) 412 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.405024+0000 mon.smithi084 (mon.0) 431 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:40.405121+0000 mon.smithi084 (mon.0) 432 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.405238+0000 mon.smithi084 (mon.0) 433 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.405583+0000 mon.smithi106 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:40.405878+0000 mon.smithi084 (mon.0) 434 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:31:41.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:40.971942+0000 mgr.smithi084.sihbqg (mgr.14162) 413 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.108329+0000 mgr.smithi084.sihbqg (mgr.14162) 411 : audit [DBG] from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:31:40.110708+0000 mgr.smithi084.sihbqg (mgr.14162) 412 : cephadm [INF] Applying service osd.None on host smithi106... 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.405024+0000 mon.smithi084 (mon.0) 431 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:40.405121+0000 mon.smithi084 (mon.0) 432 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.405238+0000 mon.smithi084 (mon.0) 433 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.405583+0000 mon.smithi106 (mon.1) 9 : audit [INF] from='osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:40.405878+0000 mon.smithi084 (mon.0) 434 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:31:41.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:40.971942+0000 mgr.smithi084.sihbqg (mgr.14162) 413 : cluster [DBG] pgmap v381: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2024-01-18T07:31:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:41.406927+0000 mon.smithi084 (mon.0) 435 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:31:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:41.407032+0000 mon.smithi084 (mon.0) 436 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-01-18T07:31:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:41.407575+0000 mon.smithi084 (mon.0) 437 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:41.410151+0000 mon.smithi084 (mon.0) 438 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:42.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:41.406927+0000 mon.smithi084 (mon.0) 435 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:31:42.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:41.407032+0000 mon.smithi084 (mon.0) 436 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-01-18T07:31:42.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:41.407575+0000 mon.smithi084 (mon.0) 437 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:42.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:41.410151+0000 mon.smithi084 (mon.0) 438 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:41.334801+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:41.334915+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:42.411113+0000 mon.smithi084 (mon.0) 439 : cluster [INF] osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] boot 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:42.411159+0000 mon.smithi084 (mon.0) 440 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:31:42.411300+0000 mon.smithi084 (mon.0) 441 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:43.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:42.972307+0000 mgr.smithi084.sihbqg (mgr.14162) 414 : cluster [DBG] pgmap v384: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:41.334801+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:41.334915+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:42.411113+0000 mon.smithi084 (mon.0) 439 : cluster [INF] osd.4 [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] boot 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:42.411159+0000 mon.smithi084 (mon.0) 440 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:31:42.411300+0000 mon.smithi084 (mon.0) 441 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-18T07:31:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:42.972307+0000 mgr.smithi084.sihbqg (mgr.14162) 414 : cluster [DBG] pgmap v384: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:44.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:43.414862+0000 mon.smithi084 (mon.0) 442 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-01-18T07:31:44.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:43.426253+0000 mon.smithi084 (mon.0) 443 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-01-18T07:31:44.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:43.414862+0000 mon.smithi084 (mon.0) 442 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-01-18T07:31:44.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:43.426253+0000 mon.smithi084 (mon.0) 443 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-01-18T07:31:45.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:44.972723+0000 mgr.smithi084.sihbqg (mgr.14162) 415 : cluster [DBG] pgmap v386: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:45.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:44.972723+0000 mgr.smithi084.sihbqg (mgr.14162) 415 : cluster [DBG] pgmap v386: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:46.973149+0000 mgr.smithi084.sihbqg (mgr.14162) 416 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:46.973149+0000 mgr.smithi084.sihbqg (mgr.14162) 416 : cluster [DBG] pgmap v387: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:48.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:46.999262+0000 mon.smithi084 (mon.0) 444 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-01-18T07:31:48.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:46.999262+0000 mon.smithi084 (mon.0) 444 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-01-18T07:31:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:48.973638+0000 mgr.smithi084.sihbqg (mgr.14162) 417 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:48.973638+0000 mgr.smithi084.sihbqg (mgr.14162) 417 : cluster [DBG] pgmap v388: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:50.974019+0000 mgr.smithi084.sihbqg (mgr.14162) 418 : cluster [DBG] pgmap v389: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:50.974019+0000 mgr.smithi084.sihbqg (mgr.14162) 418 : cluster [DBG] pgmap v389: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:52.974354+0000 mgr.smithi084.sihbqg (mgr.14162) 419 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:52.974354+0000 mgr.smithi084.sihbqg (mgr.14162) 419 : cluster [DBG] pgmap v390: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:54.974697+0000 mgr.smithi084.sihbqg (mgr.14162) 420 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:54.974697+0000 mgr.smithi084.sihbqg (mgr.14162) 420 : cluster [DBG] pgmap v391: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:56.975153+0000 mgr.smithi084.sihbqg (mgr.14162) 421 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:56.975153+0000 mgr.smithi084.sihbqg (mgr.14162) 421 : cluster [DBG] pgmap v392: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:59.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:31:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:31:58.975444+0000 mgr.smithi084.sihbqg (mgr.14162) 422 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:31:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:31:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:31:58.975444+0000 mgr.smithi084.sihbqg (mgr.14162) 422 : cluster [DBG] pgmap v393: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:00.975790+0000 mgr.smithi084.sihbqg (mgr.14162) 423 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:00.975790+0000 mgr.smithi084.sihbqg (mgr.14162) 423 : cluster [DBG] pgmap v394: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:02.976197+0000 mgr.smithi084.sihbqg (mgr.14162) 424 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:02.976197+0000 mgr.smithi084.sihbqg (mgr.14162) 424 : cluster [DBG] pgmap v395: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:04.976596+0000 mgr.smithi084.sihbqg (mgr.14162) 425 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:04.976596+0000 mgr.smithi084.sihbqg (mgr.14162) 425 : cluster [DBG] pgmap v396: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:06.976955+0000 mgr.smithi084.sihbqg (mgr.14162) 426 : cluster [DBG] pgmap v397: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:06.976955+0000 mgr.smithi084.sihbqg (mgr.14162) 426 : cluster [DBG] pgmap v397: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:08.977291+0000 mgr.smithi084.sihbqg (mgr.14162) 427 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:09.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:08.977291+0000 mgr.smithi084.sihbqg (mgr.14162) 427 : cluster [DBG] pgmap v398: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:10.977618+0000 mgr.smithi084.sihbqg (mgr.14162) 428 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:10.977618+0000 mgr.smithi084.sihbqg (mgr.14162) 428 : cluster [DBG] pgmap v399: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:12.978041+0000 mgr.smithi084.sihbqg (mgr.14162) 429 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:12.978041+0000 mgr.smithi084.sihbqg (mgr.14162) 429 : cluster [DBG] pgmap v400: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:14.978503+0000 mgr.smithi084.sihbqg (mgr.14162) 430 : cluster [DBG] pgmap v401: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:14.978503+0000 mgr.smithi084.sihbqg (mgr.14162) 430 : cluster [DBG] pgmap v401: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:16.978834+0000 mgr.smithi084.sihbqg (mgr.14162) 431 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:16.978834+0000 mgr.smithi084.sihbqg (mgr.14162) 431 : cluster [DBG] pgmap v402: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:19.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:18.979225+0000 mgr.smithi084.sihbqg (mgr.14162) 432 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:18.979225+0000 mgr.smithi084.sihbqg (mgr.14162) 432 : cluster [DBG] pgmap v403: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:20.979658+0000 mgr.smithi084.sihbqg (mgr.14162) 433 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:20.979658+0000 mgr.smithi084.sihbqg (mgr.14162) 433 : cluster [DBG] pgmap v404: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:22.979962+0000 mgr.smithi084.sihbqg (mgr.14162) 434 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:22.979962+0000 mgr.smithi084.sihbqg (mgr.14162) 434 : cluster [DBG] pgmap v405: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:25.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:24.980319+0000 mgr.smithi084.sihbqg (mgr.14162) 435 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:24.980319+0000 mgr.smithi084.sihbqg (mgr.14162) 435 : cluster [DBG] pgmap v406: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:26.980832+0000 mgr.smithi084.sihbqg (mgr.14162) 436 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:26.980832+0000 mgr.smithi084.sihbqg (mgr.14162) 436 : cluster [DBG] pgmap v407: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:28.981190+0000 mgr.smithi084.sihbqg (mgr.14162) 437 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:28.981190+0000 mgr.smithi084.sihbqg (mgr.14162) 437 : cluster [DBG] pgmap v408: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:30.981683+0000 mgr.smithi084.sihbqg (mgr.14162) 438 : cluster [DBG] pgmap v409: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:30.981683+0000 mgr.smithi084.sihbqg (mgr.14162) 438 : cluster [DBG] pgmap v409: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:32.982221+0000 mgr.smithi084.sihbqg (mgr.14162) 439 : cluster [DBG] pgmap v410: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:32.982221+0000 mgr.smithi084.sihbqg (mgr.14162) 439 : cluster [DBG] pgmap v410: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:34.982574+0000 mgr.smithi084.sihbqg (mgr.14162) 440 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:32:35.047119+0000 mon.smithi084 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:32:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:34.982574+0000 mgr.smithi084.sihbqg (mgr.14162) 440 : cluster [DBG] pgmap v411: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:32:35.047119+0000 mon.smithi084 (mon.0) 445 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:32:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:32:35.059142+0000 mon.smithi084 (mon.0) 446 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:32:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:32:35.059142+0000 mon.smithi084 (mon.0) 446 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:32:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:36.983006+0000 mgr.smithi084.sihbqg (mgr.14162) 441 : cluster [DBG] pgmap v412: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:36.983006+0000 mgr.smithi084.sihbqg (mgr.14162) 441 : cluster [DBG] pgmap v412: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:38.983482+0000 mgr.smithi084.sihbqg (mgr.14162) 442 : cluster [DBG] pgmap v413: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:38.983482+0000 mgr.smithi084.sihbqg (mgr.14162) 442 : cluster [DBG] pgmap v413: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:40.983870+0000 mgr.smithi084.sihbqg (mgr.14162) 443 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:40.983870+0000 mgr.smithi084.sihbqg (mgr.14162) 443 : cluster [DBG] pgmap v414: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:42.984255+0000 mgr.smithi084.sihbqg (mgr.14162) 444 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:42.984255+0000 mgr.smithi084.sihbqg (mgr.14162) 444 : cluster [DBG] pgmap v415: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:44.984703+0000 mgr.smithi084.sihbqg (mgr.14162) 445 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:44.984703+0000 mgr.smithi084.sihbqg (mgr.14162) 445 : cluster [DBG] pgmap v416: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:46.985256+0000 mgr.smithi084.sihbqg (mgr.14162) 446 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:46.985256+0000 mgr.smithi084.sihbqg (mgr.14162) 446 : cluster [DBG] pgmap v417: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:48.985736+0000 mgr.smithi084.sihbqg (mgr.14162) 447 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:48.985736+0000 mgr.smithi084.sihbqg (mgr.14162) 447 : cluster [DBG] pgmap v418: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:50.986178+0000 mgr.smithi084.sihbqg (mgr.14162) 448 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:50.986178+0000 mgr.smithi084.sihbqg (mgr.14162) 448 : cluster [DBG] pgmap v419: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:52.986579+0000 mgr.smithi084.sihbqg (mgr.14162) 449 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:52.986579+0000 mgr.smithi084.sihbqg (mgr.14162) 449 : cluster [DBG] pgmap v420: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:54.986979+0000 mgr.smithi084.sihbqg (mgr.14162) 450 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:54.986979+0000 mgr.smithi084.sihbqg (mgr.14162) 450 : cluster [DBG] pgmap v421: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:56.987380+0000 mgr.smithi084.sihbqg (mgr.14162) 451 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:56.987380+0000 mgr.smithi084.sihbqg (mgr.14162) 451 : cluster [DBG] pgmap v422: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:32:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:32:58.987719+0000 mgr.smithi084.sihbqg (mgr.14162) 452 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:32:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:32:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:32:58.987719+0000 mgr.smithi084.sihbqg (mgr.14162) 452 : cluster [DBG] pgmap v423: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:00.988156+0000 mgr.smithi084.sihbqg (mgr.14162) 453 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:01.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:00.988156+0000 mgr.smithi084.sihbqg (mgr.14162) 453 : cluster [DBG] pgmap v424: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:02.988527+0000 mgr.smithi084.sihbqg (mgr.14162) 454 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:02.988527+0000 mgr.smithi084.sihbqg (mgr.14162) 454 : cluster [DBG] pgmap v425: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:04.988978+0000 mgr.smithi084.sihbqg (mgr.14162) 455 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:04.988978+0000 mgr.smithi084.sihbqg (mgr.14162) 455 : cluster [DBG] pgmap v426: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:06.989415+0000 mgr.smithi084.sihbqg (mgr.14162) 456 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:06.989415+0000 mgr.smithi084.sihbqg (mgr.14162) 456 : cluster [DBG] pgmap v427: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:08.989946+0000 mgr.smithi084.sihbqg (mgr.14162) 457 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:08.989946+0000 mgr.smithi084.sihbqg (mgr.14162) 457 : cluster [DBG] pgmap v428: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:10.990426+0000 mgr.smithi084.sihbqg (mgr.14162) 458 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:10.990426+0000 mgr.smithi084.sihbqg (mgr.14162) 458 : cluster [DBG] pgmap v429: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:12.990941+0000 mgr.smithi084.sihbqg (mgr.14162) 459 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:12.990941+0000 mgr.smithi084.sihbqg (mgr.14162) 459 : cluster [DBG] pgmap v430: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:14.991270+0000 mgr.smithi084.sihbqg (mgr.14162) 460 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:14.991270+0000 mgr.smithi084.sihbqg (mgr.14162) 460 : cluster [DBG] pgmap v431: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:17.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:16.991689+0000 mgr.smithi084.sihbqg (mgr.14162) 461 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:16.991689+0000 mgr.smithi084.sihbqg (mgr.14162) 461 : cluster [DBG] pgmap v432: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:18.992108+0000 mgr.smithi084.sihbqg (mgr.14162) 462 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:18.992108+0000 mgr.smithi084.sihbqg (mgr.14162) 462 : cluster [DBG] pgmap v433: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:20.992476+0000 mgr.smithi084.sihbqg (mgr.14162) 463 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:20.992476+0000 mgr.smithi084.sihbqg (mgr.14162) 463 : cluster [DBG] pgmap v434: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:22.992873+0000 mgr.smithi084.sihbqg (mgr.14162) 464 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:22.992873+0000 mgr.smithi084.sihbqg (mgr.14162) 464 : cluster [DBG] pgmap v435: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:24.993349+0000 mgr.smithi084.sihbqg (mgr.14162) 465 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:24.993349+0000 mgr.smithi084.sihbqg (mgr.14162) 465 : cluster [DBG] pgmap v436: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:26.993736+0000 mgr.smithi084.sihbqg (mgr.14162) 466 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:26.993736+0000 mgr.smithi084.sihbqg (mgr.14162) 466 : cluster [DBG] pgmap v437: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:28.994250+0000 mgr.smithi084.sihbqg (mgr.14162) 467 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:28.994250+0000 mgr.smithi084.sihbqg (mgr.14162) 467 : cluster [DBG] pgmap v438: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:30.994772+0000 mgr.smithi084.sihbqg (mgr.14162) 468 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:30.994772+0000 mgr.smithi084.sihbqg (mgr.14162) 468 : cluster [DBG] pgmap v439: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:32.995129+0000 mgr.smithi084.sihbqg (mgr.14162) 469 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:32.995129+0000 mgr.smithi084.sihbqg (mgr.14162) 469 : cluster [DBG] pgmap v440: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:34.995601+0000 mgr.smithi084.sihbqg (mgr.14162) 470 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:35.049928+0000 mon.smithi084 (mon.0) 447 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:33:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:35.060411+0000 mon.smithi084 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:33:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:34.995601+0000 mgr.smithi084.sihbqg (mgr.14162) 470 : cluster [DBG] pgmap v441: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:35.049928+0000 mon.smithi084 (mon.0) 447 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:33:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:35.060411+0000 mon.smithi084 (mon.0) 448 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:33:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:36.995997+0000 mgr.smithi084.sihbqg (mgr.14162) 471 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:36.995997+0000 mgr.smithi084.sihbqg (mgr.14162) 471 : cluster [DBG] pgmap v442: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:33:37.953702+0000 mgr.smithi084.sihbqg (mgr.14162) 472 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:33:38.194 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:33:38.195 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:33:38.196 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:33:38.197 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:33:38.197 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:33:38.197 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.197 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:37.956174+0000 mon.smithi084 (mon.0) 449 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-18T07:33:38.197 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:37.956616+0000 mgr.smithi084.sihbqg (mgr.14162) 473 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:33:37.953702+0000 mgr.smithi084.sihbqg (mgr.14162) 472 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:33:38.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:33:38.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:37.956174+0000 mon.smithi084 (mon.0) 449 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-18T07:33:38.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:37.956616+0000 mgr.smithi084.sihbqg (mgr.14162) 473 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:38.841259+0000 mon.smithi106 (mon.1) 10 : audit [INF] from='client.? 172.21.15.106:0/3530701891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]: dispatch 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:38.841944+0000 mon.smithi084 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]: dispatch 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:38.846259+0000 mon.smithi084 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]': finished 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:38.846323+0000 mon.smithi084 (mon.0) 452 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:38.846454+0000 mon.smithi084 (mon.0) 453 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:38.996318+0000 mgr.smithi084.sihbqg (mgr.14162) 474 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:38.841259+0000 mon.smithi106 (mon.1) 10 : audit [INF] from='client.? 172.21.15.106:0/3530701891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]: dispatch 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:38.841944+0000 mon.smithi084 (mon.0) 450 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]: dispatch 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:38.846259+0000 mon.smithi084 (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c59b868-d177-4948-9e86-cb37faa7ce12"}]': finished 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:38.846323+0000 mon.smithi084 (mon.0) 452 : cluster [DBG] osdmap e33: 6 total, 5 up, 6 in 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:38.846454+0000 mon.smithi084 (mon.0) 453 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:39.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:38.996318+0000 mgr.smithi084.sihbqg (mgr.14162) 474 : cluster [DBG] pgmap v444: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:39.252140+0000 mon.smithi106 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.106:0/54532968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:33:40.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:39.252140+0000 mon.smithi106 (mon.1) 11 : audit [DBG] from='client.? 172.21.15.106:0/54532968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-18T07:33:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:40.996645+0000 mgr.smithi084.sihbqg (mgr.14162) 475 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:40.996645+0000 mgr.smithi084.sihbqg (mgr.14162) 475 : cluster [DBG] pgmap v445: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:42.997167+0000 mgr.smithi084.sihbqg (mgr.14162) 476 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:42.997167+0000 mgr.smithi084.sihbqg (mgr.14162) 476 : cluster [DBG] pgmap v446: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:44.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:44.607759+0000 mon.smithi084 (mon.0) 454 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-18T07:33:44.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:44.608493+0000 mon.smithi084 (mon.0) 455 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:33:45.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:44.607759+0000 mon.smithi084 (mon.0) 454 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-18T07:33:45.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:44.608493+0000 mon.smithi084 (mon.0) 455 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-18T07:33:45.909 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:33:44.609069+0000 mgr.smithi084.sihbqg (mgr.14162) 477 : cephadm [INF] Deploying daemon osd.5 on smithi106 2024-01-18T07:33:45.909 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:44.997632+0000 mgr.smithi084.sihbqg (mgr.14162) 478 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:46.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:33:44.609069+0000 mgr.smithi084.sihbqg (mgr.14162) 477 : cephadm [INF] Deploying daemon osd.5 on smithi106 2024-01-18T07:33:46.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:44.997632+0000 mgr.smithi084.sihbqg (mgr.14162) 478 : cluster [DBG] pgmap v447: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:46.928 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 5 on host 'smithi106' 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.926+0000 7ff59dffb700 1 -- 172.21.15.106:0/2026834717 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff5a80fdee0 con 0x7ff59405c2b0 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 msgr2=0x7ff59405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 0x7ff59405e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff5a8105ba0 tx=0x7ff59800b040).stop 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 msgr2=0x7ff5a8106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff5a4000c00 tx=0x7ff5a400b040).stop 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 shutdown_connections 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff5a80f55c0 0x7ff5a8105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff59405c2b0 0x7ff59405e770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 --2- 172.21.15.106:0/2026834717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5a80f6860 0x7ff5a8106320 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 >> 172.21.15.106:0/2026834717 conn(0x7ff5a80f0d40 msgr2=0x7ff5a80fa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 shutdown_connections 2024-01-18T07:33:46.930 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:46.928+0000 7ff5af542700 1 -- 172.21.15.106:0/2026834717 wait complete. 2024-01-18T07:33:47.258 DEBUG:teuthology.orchestra.run.smithi106:osd.5> sudo journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.5.service 2024-01-18T07:33:47.261 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-01-18T07:33:47.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd stat -f json 2024-01-18T07:33:47.291 INFO:journalctl@ceph.osd.5.smithi106.stdout:-- Logs begin at Thu 2024-01-18 07:09:06 UTC. -- 2024-01-18T07:33:47.664 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:33:47.381+0000 7fe078e3a080 -1 Falling back to public interface 2024-01-18T07:33:48.118 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.923099+0000 mon.smithi084 (mon.0) 456 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.118 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.927122+0000 mon.smithi084 (mon.0) 457 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.969854+0000 mon.smithi084 (mon.0) 458 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.84:9093"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.970206+0000 mgr.smithi084.sihbqg (mgr.14162) 479 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.84:9093"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.982039+0000 mon.smithi084 (mon.0) 459 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.985663+0000 mon.smithi084 (mon.0) 460 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.986006+0000 mgr.smithi084.sihbqg (mgr.14162) 480 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.990515+0000 mon.smithi084 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.84:3000"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.990794+0000 mgr.smithi084.sihbqg (mgr.14162) 481 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.84:3000"}]: dispatch 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.996688+0000 mon.smithi084 (mon.0) 462 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.119 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:46.998050+0000 mgr.smithi084.sihbqg (mgr.14162) 482 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:48.120 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:46.998921+0000 mon.smithi084 (mon.0) 463 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:33:48.120 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:47.222889+0000 mon.smithi084 (mon.0) 464 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.120 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:47.331668+0000 mon.smithi084 (mon.0) 465 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.923099+0000 mon.smithi084 (mon.0) 456 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.927122+0000 mon.smithi084 (mon.0) 457 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.969854+0000 mon.smithi084 (mon.0) 458 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.84:9093"}]: dispatch 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.970206+0000 mgr.smithi084.sihbqg (mgr.14162) 479 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.84:9093"}]: dispatch 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.982039+0000 mon.smithi084 (mon.0) 459 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.985663+0000 mon.smithi084 (mon.0) 460 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-18T07:33:48.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.986006+0000 mgr.smithi084.sihbqg (mgr.14162) 480 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.990515+0000 mon.smithi084 (mon.0) 461 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.84:3000"}]: dispatch 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.990794+0000 mgr.smithi084.sihbqg (mgr.14162) 481 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.84:3000"}]: dispatch 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.996688+0000 mon.smithi084 (mon.0) 462 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:46.998050+0000 mgr.smithi084.sihbqg (mgr.14162) 482 : cluster [DBG] pgmap v448: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:46.998921+0000 mon.smithi084 (mon.0) 463 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:47.222889+0000 mon.smithi084 (mon.0) 464 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:48.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:47.331668+0000 mon.smithi084 (mon.0) 465 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:49.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/189749789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c0060310 msgr2=0x7ff9c00f9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/189749789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c0060310 0x7ff9c00f9110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff9b0004660 tx=0x7ff9b0009b30).stop 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/189749789 shutdown_connections 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/189749789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c00f9690 0x7ff9c00fbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/189749789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c0060310 0x7ff9c00f9110 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/189749789 >> 172.21.15.84:0/189749789 conn(0x7ff9c00f0b90 msgr2=0x7ff9c00f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/189749789 shutdown_connections 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.012+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/189749789 wait complete. 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 Processor -- start 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 -- start start 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c0060310 0x7ff9c0106d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:49.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c0107850 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9c4ad2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c0108060 con 0x7ff9c0060310 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9beffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9beffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59084/0 (socket says 172.21.15.84:59084) 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9beffd700 1 -- 172.21.15.84:0/2404547622 learned_addr learned my addr 172.21.15.84:0/2404547622 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.013+0000 7ff9bf7fe700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c0060310 0x7ff9c0106d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9beffd700 1 -- 172.21.15.84:0/2404547622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c0060310 msgr2=0x7ff9c0106d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9beffd700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c0060310 0x7ff9c0106d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9beffd700 1 -- 172.21.15.84:0/2404547622 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b0005040 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9beffd700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7ff9b40026d0 tx=0x7ff9b4009e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9b40079c0 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9b400f040 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c01082e0 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9b4014850 con 0x7ff9c00f9690 2024-01-18T07:33:49.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.014+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c01088a0 con 0x7ff9c00f9690 2024-01-18T07:33:49.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.015+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c0059100 con 0x7ff9c00f9690 2024-01-18T07:33:49.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.015+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff9b4007b20 con 0x7ff9c00f9690 2024-01-18T07:33:49.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.016+0000 7ff9a7fff700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 0x7ff9a805e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:49.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.016+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4235+0+0 (secure 0 0 0) 0x7ff9b4075a40 con 0x7ff9c00f9690 2024-01-18T07:33:49.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.016+0000 7ff9bf7fe700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 0x7ff9a805e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:49.018 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.016+0000 7ff9bf7fe700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 0x7ff9a805e7c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff9c0106ad0 tx=0x7ff9b0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:49.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.020+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9b4048b80 con 0x7ff9c00f9690 2024-01-18T07:33:49.174 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.173+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff9c00fee70 con 0x7ff9c00f9690 2024-01-18T07:33:49.174 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.174+0000 7ff9a7fff700 1 -- 172.21.15.84:0/2404547622 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v33) v1 ==== 74+0+130 (secure 0 0 0) 0x7ff9c00fee70 con 0x7ff9c00f9690 2024-01-18T07:33:49.174 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:33:49.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.176+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 msgr2=0x7ff9a805e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:49.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.176+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 0x7ff9a805e7c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff9c0106ad0 tx=0x7ff9b0009b30).stop 2024-01-18T07:33:49.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.176+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 msgr2=0x7ff9c0107250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:49.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.176+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7ff9b40026d0 tx=0x7ff9b4009e00).stop 2024-01-18T07:33:49.177 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 shutdown_connections 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff9a805c300 0x7ff9a805e7c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff9c0060310 0x7ff9c0106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 --2- 172.21.15.84:0/2404547622 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff9c00f9690 0x7ff9c0107250 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 >> 172.21.15.84:0/2404547622 conn(0x7ff9c00f0b90 msgr2=0x7ff9c00f2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 shutdown_connections 2024-01-18T07:33:49.178 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:49.177+0000 7ff9c4ad2700 1 -- 172.21.15.84:0/2404547622 wait complete. 2024-01-18T07:33:49.521 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:48.370042+0000 mon.smithi084 (mon.0) 466 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:49.521 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:48.998347+0000 mgr.smithi084.sihbqg (mgr.14162) 483 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:49.521 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:49.174885+0000 mon.smithi084 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.84:0/2404547622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:49.547 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":33,"num_osds":6,"num_up_osds":5,"osd_up_since":1705563102,"num_in_osds":6,"osd_in_since":1705563218,"num_remapped_pgs":0} 2024-01-18T07:33:49.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:48.370042+0000 mon.smithi084 (mon.0) 466 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:49.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:48.998347+0000 mgr.smithi084.sihbqg (mgr.14162) 483 : cluster [DBG] pgmap v449: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:49.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:49.174885+0000 mon.smithi084 (mon.0) 467 : audit [DBG] from='client.? 172.21.15.84:0/2404547622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:50.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd stat -f json 2024-01-18T07:33:51.164 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:33:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:33:50.878+0000 7fe078e3a080 -1 osd.5 0 log_to_monitors {default=true} 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:50.402675+0000 mon.smithi084 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:33:50.782790+0000 mgr.smithi084.sihbqg (mgr.14162) 484 : cephadm [INF] Detected new or changed devices on smithi106 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:50.787605+0000 mon.smithi084 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:50.882642+0000 mon.smithi106 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:50.883212+0000 mon.smithi084 (mon.0) 470 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-18T07:33:51.660 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:50.998673+0000 mgr.smithi084.sihbqg (mgr.14162) 485 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:50.402675+0000 mon.smithi084 (mon.0) 468 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:33:50.782790+0000 mgr.smithi084.sihbqg (mgr.14162) 484 : cephadm [INF] Detected new or changed devices on smithi106 2024-01-18T07:33:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:50.787605+0000 mon.smithi084 (mon.0) 469 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:50.882642+0000 mon.smithi106 (mon.1) 12 : audit [INF] from='osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-18T07:33:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:50.883212+0000 mon.smithi084 (mon.0) 470 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-18T07:33:51.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:50.998673+0000 mgr.smithi084.sihbqg (mgr.14162) 485 : cluster [DBG] pgmap v450: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 -- 172.21.15.84:0/806320610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 msgr2=0x7f854c0fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 --2- 172.21.15.84:0/806320610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c0fdfe0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f853c004660 tx=0x7f853c009b30).stop 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 -- 172.21.15.84:0/806320610 shutdown_connections 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 --2- 172.21.15.84:0/806320610 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c0fdfe0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 --2- 172.21.15.84:0/806320610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c0fb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 -- 172.21.15.84:0/806320610 >> 172.21.15.84:0/806320610 conn(0x7f854c0f0d40 msgr2=0x7f854c0f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:52.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 -- 172.21.15.84:0/806320610 shutdown_connections 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.246+0000 7f8554444700 1 -- 172.21.15.84:0/806320610 wait complete. 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 Processor -- start 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 -- start start 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c109160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c109c00 con 0x7f854c0fbbf0 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f8554444700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c109d40 con 0x7f854c0602b0 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59094/0 (socket says 172.21.15.84:59094) 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 -- 172.21.15.84:0/2846334679 learned_addr learned my addr 172.21.15.84:0/2846334679 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85521e0700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c109160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 -- 172.21.15.84:0/2846334679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 msgr2=0x7f854c109160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c109160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 -- 172.21.15.84:0/2846334679 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c005040 con 0x7f854c0fbbf0 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85521e0700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.247+0000 7f85519df700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f853c004e30 tx=0x7f853c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.248+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853c018070 con 0x7f854c0fbbf0 2024-01-18T07:33:52.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.248+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c10a680 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.248+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f853c013410 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.248+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f853c01c3c0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.248+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c10abe0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.249+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c045ad0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.249+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f853c01c680 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.249+0000 7f8542ffd700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 0x7f853805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.250+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4256+0+0 (secure 0 0 0) 0x7f853c077670 con 0x7f854c0fbbf0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.250+0000 7f85521e0700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 0x7f853805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:52.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.250+0000 7f85521e0700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 0x7f853805e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8548009740 tx=0x7f8548009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:52.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.253+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f853c04a7a0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.411+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f854c1010f0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.411+0000 7f8542ffd700 1 -- 172.21.15.84:0/2846334679 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v34) v1 ==== 74+0+130 (secure 0 0 0) 0x7f853c04ddc0 con 0x7f854c0fbbf0 2024-01-18T07:33:52.412 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:33:52.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 msgr2=0x7f853805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:52.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 0x7f853805e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8548009740 tx=0x7f8548009be0).stop 2024-01-18T07:33:52.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 msgr2=0x7f854c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f853c004e30 tx=0x7f853c009b30).stop 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 shutdown_connections 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f853805c2b0 0x7f853805e770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f854c0602b0 0x7f854c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 --2- 172.21.15.84:0/2846334679 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f854c0fbbf0 0x7f854c109650 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 >> 172.21.15.84:0/2846334679 conn(0x7f854c0f0d40 msgr2=0x7f854c0f3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 shutdown_connections 2024-01-18T07:33:52.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:52.413+0000 7f8554444700 1 -- 172.21.15.84:0/2846334679 wait complete. 2024-01-18T07:33:52.832 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":34,"num_osds":6,"num_up_osds":5,"osd_up_since":1705563102,"num_in_osds":6,"osd_in_since":1705563218,"num_remapped_pgs":0} 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:51.796179+0000 mon.smithi084 (mon.0) 471 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:51.796258+0000 mon.smithi084 (mon.0) 472 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:51.796921+0000 mon.smithi084 (mon.0) 473 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:51.799314+0000 mon.smithi106 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:51.799694+0000 mon.smithi084 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:33:53.050 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:52.412272+0000 mon.smithi084 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.84:0/2846334679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:53.164 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:33:52.802+0000 7fe06f3ab700 -1 osd.5 0 waiting for initial osdmap 2024-01-18T07:33:53.164 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:33:52.812+0000 7fe06b326700 -1 osd.5 35 set_numa_affinity unable to identify public interface 'enp3s0f1' numa node: (0) Success 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:51.796179+0000 mon.smithi084 (mon.0) 471 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:51.796258+0000 mon.smithi084 (mon.0) 472 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:51.796921+0000 mon.smithi084 (mon.0) 473 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:51.799314+0000 mon.smithi106 (mon.1) 13 : audit [INF] from='osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:51.799694+0000 mon.smithi084 (mon.0) 474 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-01-18T07:33:53.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:52.412272+0000 mon.smithi084 (mon.0) 475 : audit [DBG] from='client.? 172.21.15.84:0/2846334679' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:53.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd stat -f json 2024-01-18T07:33:54.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:52.798170+0000 mon.smithi084 (mon.0) 476 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:52.798255+0000 mon.smithi084 (mon.0) 477 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:52.798617+0000 mon.smithi084 (mon.0) 478 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:52.802183+0000 mon.smithi084 (mon.0) 479 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:52.998966+0000 mgr.smithi084.sihbqg (mgr.14162) 486 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:53.439352+0000 mon.smithi084 (mon.0) 480 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:54.133 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:53.681260+0000 mon.smithi084 (mon.0) 481 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:54.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:52.798170+0000 mon.smithi084 (mon.0) 476 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-01-18T07:33:54.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:52.798255+0000 mon.smithi084 (mon.0) 477 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-01-18T07:33:54.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:52.798617+0000 mon.smithi084 (mon.0) 478 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:54.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:52.802183+0000 mon.smithi084 (mon.0) 479 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:54.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:52.998966+0000 mgr.smithi084.sihbqg (mgr.14162) 486 : cluster [DBG] pgmap v453: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2024-01-18T07:33:54.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:53.439352+0000 mon.smithi084 (mon.0) 480 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:54.165 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:53.681260+0000 mon.smithi084 (mon.0) 481 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:33:55.050 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 -- 172.21.15.84:0/2213292789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f7970 msgr2=0x7fa2e80f7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/2213292789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f7970 0x7fa2e80f7dd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fa2d0004660 tx=0x7fa2d0009b30).stop 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 -- 172.21.15.84:0/2213292789 shutdown_connections 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/2213292789 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f7970 0x7fa2e80f7dd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/2213292789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f66d0 0x7fa2e80f6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 -- 172.21.15.84:0/2213292789 >> 172.21.15.84:0/2213292789 conn(0x7fa2e80f1e10 msgr2=0x7fa2e80f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 -- 172.21.15.84:0/2213292789 shutdown_connections 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.048+0000 7fa2ed81a700 1 -- 172.21.15.84:0/2213292789 wait complete. 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 Processor -- start 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 -- start start 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:55.051 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 0x7fa2e8107440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81079f0 con 0x7fa2e80f66d0 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2ed81a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8107b30 con 0x7fa2e80f7970 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59102/0 (socket says 172.21.15.84:59102) 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 -- 172.21.15.84:0/3077511686 learned_addr learned my addr 172.21.15.84:0/3077511686 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e67fc700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 0x7fa2e8107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 -- 172.21.15.84:0/3077511686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 msgr2=0x7fa2e8107440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 0x7fa2e8107440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 -- 172.21.15.84:0/3077511686 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0005040 con 0x7fa2e80f66d0 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e67fc700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 0x7fa2e8107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.049+0000 7fa2e6ffd700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa2d800c530 tx=0x7fa2d8007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:55.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d800f650 con 0x7fa2e80f66d0 2024-01-18T07:33:55.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e81084d0 con 0x7fa2e80f66d0 2024-01-18T07:33:55.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2d800fc60 con 0x7fa2e80f66d0 2024-01-18T07:33:55.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2d8012d20 con 0x7fa2e80f66d0 2024-01-18T07:33:55.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8108a90 con 0x7fa2e80f66d0 2024-01-18T07:33:55.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.050+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8045ad0 con 0x7fa2e80f66d0 2024-01-18T07:33:55.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.051+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa2d8019030 con 0x7fa2e80f66d0 2024-01-18T07:33:55.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.051+0000 7fa2df7fe700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 0x7fa2d405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:55.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.051+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa2d8075830 con 0x7fa2e80f66d0 2024-01-18T07:33:55.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.051+0000 7fa2e67fc700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 0x7fa2d405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:55.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.052+0000 7fa2e67fc700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 0x7fa2d405e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa2d0013580 tx=0x7fa2d0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:55.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.055+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2d8048880 con 0x7fa2e80f66d0 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:51.897939+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:51.898059+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:33:53.685121+0000 mgr.smithi084.sihbqg (mgr.14162) 487 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:53.803852+0000 mon.smithi084 (mon.0) 482 : cluster [INF] osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] boot 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:53.803889+0000 mon.smithi084 (mon.0) 483 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-01-18T07:33:55.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:53.804281+0000 mon.smithi084 (mon.0) 484 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:51.897939+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:51.898059+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:33:53.685121+0000 mgr.smithi084.sihbqg (mgr.14162) 487 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:53.803852+0000 mon.smithi084 (mon.0) 482 : cluster [INF] osd.5 [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] boot 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:53.803889+0000 mon.smithi084 (mon.0) 483 : cluster [DBG] osdmap e36: 6 total, 6 up, 6 in 2024-01-18T07:33:55.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:53.804281+0000 mon.smithi084 (mon.0) 484 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-18T07:33:55.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.209+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa2e80fefe0 con 0x7fa2e80f66d0 2024-01-18T07:33:55.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.209+0000 7fa2df7fe700 1 -- 172.21.15.84:0/3077511686 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa2d8017070 con 0x7fa2e80f66d0 2024-01-18T07:33:55.210 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 msgr2=0x7fa2d405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 0x7fa2d405e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa2d0013580 tx=0x7fa2d0009b30).stop 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 msgr2=0x7fa2e8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa2d800c530 tx=0x7fa2d8007e10).stop 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 shutdown_connections 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa2e80f66d0 0x7fa2e8106f50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa2d405c2b0 0x7fa2d405e770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 --2- 172.21.15.84:0/3077511686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2e80f7970 0x7fa2e8107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 >> 172.21.15.84:0/3077511686 conn(0x7fa2e80f1e10 msgr2=0x7fa2e80fb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 shutdown_connections 2024-01-18T07:33:55.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:55.211+0000 7fa2ed81a700 1 -- 172.21.15.84:0/3077511686 wait complete. 2024-01-18T07:33:55.630 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"num_osds":6,"num_up_osds":6,"osd_up_since":1705563233,"num_in_osds":6,"osd_in_since":1705563218,"num_remapped_pgs":0} 2024-01-18T07:33:55.631 INFO:tasks.cephadm:Setting up client nodes... 2024-01-18T07:33:55.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-18T07:33:55.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:54.813718+0000 mon.smithi084 (mon.0) 485 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-01-18T07:33:55.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:54.999269+0000 mgr.smithi084.sihbqg (mgr.14162) 488 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:33:55.883 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:55.210589+0000 mon.smithi084 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.84:0/3077511686' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:56.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:54.813718+0000 mon.smithi084 (mon.0) 485 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-01-18T07:33:56.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:54.999269+0000 mgr.smithi084.sihbqg (mgr.14162) 488 : cluster [DBG] pgmap v456: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:33:56.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:55.210589+0000 mon.smithi084 (mon.0) 486 : audit [DBG] from='client.? 172.21.15.84:0/3077511686' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-18T07:33:56.737 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 -- 172.21.15.84:0/1592065615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 msgr2=0x7ff8f40f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1592065615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f40f7e40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7ff8e4004660 tx=0x7ff8e4009b30).stop 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 -- 172.21.15.84:0/1592065615 shutdown_connections 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1592065615 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f40f7e40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1592065615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 0x7ff8f40f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 -- 172.21.15.84:0/1592065615 >> 172.21.15.84:0/1592065615 conn(0x7ff8f40f1e80 msgr2=0x7ff8f40f42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 -- 172.21.15.84:0/1592065615 shutdown_connections 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.735+0000 7ff8fa267700 1 -- 172.21.15.84:0/1592065615 wait complete. 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 Processor -- start 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 -- start start 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 0x7ff8f406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f406a380 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8fa267700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f406a4c0 con 0x7ff8f40f6740 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57742/0 (socket says 172.21.15.84:57742) 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 -- 172.21.15.84:0/1617238889 learned_addr learned my addr 172.21.15.84:0/1617238889 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f3fff700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 0x7ff8f406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 -- 172.21.15.84:0/1617238889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 msgr2=0x7ff8f406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 0x7ff8f406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.736+0000 7ff8f37fe700 1 -- 172.21.15.84:0/1617238889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4005040 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8f37fe700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff8e4009f50 tx=0x7ff8e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e4018070 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff8e4013410 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8e401c3c0 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f406a740 con 0x7ff8f40f79e0 2024-01-18T07:33:56.739 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.737+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f406aca0 con 0x7ff8f40f79e0 2024-01-18T07:33:56.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.738+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f4045ad0 con 0x7ff8f40f79e0 2024-01-18T07:33:56.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.738+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff8e401d420 con 0x7ff8f40f79e0 2024-01-18T07:33:56.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.738+0000 7ff8f0ff9700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 0x7ff8e005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:56.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.738+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff8e4077750 con 0x7ff8f40f79e0 2024-01-18T07:33:56.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.739+0000 7ff8f3fff700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 0x7ff8e005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:56.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.739+0000 7ff8f3fff700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 0x7ff8e005e770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff8f406b730 tx=0x7ff8dc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:56.743 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.743+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8e404a7d0 con 0x7ff8f40f79e0 2024-01-18T07:33:56.948 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.947+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7ff8f4027a00 con 0x7ff8f40f79e0 2024-01-18T07:33:56.951 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.951+0000 7ff8f0ff9700 1 -- 172.21.15.84:0/1617238889 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7ff8e404ddf0 con 0x7ff8f40f79e0 2024-01-18T07:33:56.952 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-01-18T07:33:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQBk1KhlueeSOBAAuHyjPriJ9KIIcRvwpFXWCQ== 2024-01-18T07:33:56.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 msgr2=0x7ff8e005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:56.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 0x7ff8e005e770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7ff8f406b730 tx=0x7ff8dc00b040).stop 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 msgr2=0x7ff8f4069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff8e4009f50 tx=0x7ff8e4009b30).stop 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 shutdown_connections 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff8e005c2b0 0x7ff8e005e770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8f40f6740 0x7ff8f406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 --2- 172.21.15.84:0/1617238889 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff8f40f79e0 0x7ff8f4069e20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 >> 172.21.15.84:0/1617238889 conn(0x7ff8f40f1e80 msgr2=0x7ff8f40fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 shutdown_connections 2024-01-18T07:33:56.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:33:56.953+0000 7ff8fa267700 1 -- 172.21.15.84:0/1617238889 wait complete. 2024-01-18T07:33:57.268 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:56.949005+0000 mon.smithi084 (mon.0) 487 : audit [INF] from='client.? 172.21.15.84:0/1617238889' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:57.268 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:56.952364+0000 mon.smithi084 (mon.0) 488 : audit [INF] from='client.? 172.21.15.84:0/1617238889' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-18T07:33:57.269 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:56.999591+0000 mgr.smithi084.sihbqg (mgr.14162) 489 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:33:57.327 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-01-18T07:33:57.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-01-18T07:33:57.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-18T07:33:57.366 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-01-18T07:33:57.400 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:56.949005+0000 mon.smithi084 (mon.0) 487 : audit [INF] from='client.? 172.21.15.84:0/1617238889' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:57.400 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:56.952364+0000 mon.smithi084 (mon.0) 488 : audit [INF] from='client.? 172.21.15.84:0/1617238889' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-18T07:33:57.400 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:56.999591+0000 mgr.smithi084.sihbqg (mgr.14162) 489 : cluster [DBG] pgmap v457: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:33:58.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- 172.21.15.106:0/815502313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 msgr2=0x7f8b640f7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:58.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 --2- 172.21.15.106:0/815502313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b640f7c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f8b60004660 tx=0x7f8b60009b30).stop 2024-01-18T07:33:58.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- 172.21.15.106:0/815502313 shutdown_connections 2024-01-18T07:33:58.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 --2- 172.21.15.106:0/815502313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b640f7c30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 --2- 172.21.15.106:0/815502313 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b640f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- 172.21.15.106:0/815502313 >> 172.21.15.106:0/815502313 conn(0x7f8b640f1c90 msgr2=0x7f8b640f40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- 172.21.15.106:0/815502313 shutdown_connections 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- 172.21.15.106:0/815502313 wait complete. 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 Processor -- start 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.517+0000 7f8b6b562700 1 -- start start 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6406a380 con 0x7f8b640f6530 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6406a4c0 con 0x7f8b640f77d0 2024-01-18T07:33:58.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:35140/0 (socket says 172.21.15.106:35140) 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 -- 172.21.15.106:0/305566835 learned_addr learned my addr 172.21.15.106:0/305566835 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 -- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 msgr2=0x7f8b6406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6a560700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b6406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 -- 172.21.15.106:0/305566835 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60005040 con 0x7f8b640f77d0 2024-01-18T07:33:58.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6a560700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b6406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b69d5f700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8b60009830 tx=0x7f8b60009b30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b60018070 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b60013410 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6406a740 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b6001c620 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.518+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6406aca0 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.519+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b64045ad0 con 0x7f8b640f77d0 2024-01-18T07:33:58.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.519+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8b6001c890 con 0x7f8b640f77d0 2024-01-18T07:33:58.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.520+0000 7f8b5affd700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 0x7f8b5005e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:33:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.520+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8b60077970 con 0x7f8b640f77d0 2024-01-18T07:33:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.520+0000 7f8b6a560700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 0x7f8b5005e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:33:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.520+0000 7f8b6a560700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 0x7f8b5005e7c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f8b6406b730 tx=0x7f8b54009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:33:58.525 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.524+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b6004a990 con 0x7f8b640f77d0 2024-01-18T07:33:58.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.729+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8b64027a00 con 0x7f8b640f77d0 2024-01-18T07:33:58.735 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.734+0000 7f8b5affd700 1 -- 172.21.15.106:0/305566835 <== mon.1 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8b6004dfb0 con 0x7f8b640f77d0 2024-01-18T07:33:58.735 INFO:teuthology.orchestra.run.smithi106.stdout:[client.1] 2024-01-18T07:33:58.735 INFO:teuthology.orchestra.run.smithi106.stdout: key = AQBm1KhlS2ahKxAAmBl3GWc9yvLnmoKMYsrn7A== 2024-01-18T07:33:58.736 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 msgr2=0x7f8b5005e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 0x7f8b5005e7c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f8b6406b730 tx=0x7f8b54009040).stop 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 msgr2=0x7f8b64069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8b60009830 tx=0x7f8b60009b30).stop 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 shutdown_connections 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8b640f6530 0x7f8b6406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8b5005c300 0x7f8b5005e7c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 --2- 172.21.15.106:0/305566835 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b640f77d0 0x7f8b64069e20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 >> 172.21.15.106:0/305566835 conn(0x7f8b640f1c90 msgr2=0x7f8b640fb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 shutdown_connections 2024-01-18T07:33:58.737 INFO:teuthology.orchestra.run.smithi106.stderr:2024-01-18T07:33:58.735+0000 7f8b6b562700 1 -- 172.21.15.106:0/305566835 wait complete. 2024-01-18T07:33:59.038 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:58.731091+0000 mon.smithi106 (mon.1) 14 : audit [INF] from='client.? 172.21.15.106:0/305566835' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:59.038 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:58.731833+0000 mon.smithi084 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:59.038 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:33:58.735238+0000 mon.smithi084 (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-18T07:33:59.114 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-01-18T07:33:59.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-01-18T07:33:59.114 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-18T07:33:59.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-18T07:33:59.224 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:58.731091+0000 mon.smithi106 (mon.1) 14 : audit [INF] from='client.? 172.21.15.106:0/305566835' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:59.224 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:58.731833+0000 mon.smithi084 (mon.0) 489 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-01-18T07:33:59.224 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:33:58.735238+0000 mon.smithi084 (mon.0) 490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-01-18T07:33:59.425 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:00.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:33:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:33:59.000211+0000 mgr.smithi084.sihbqg (mgr.14162) 490 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:00.385 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- 172.21.15.84:0/4121324991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 msgr2=0x7f79700fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 --2- 172.21.15.84:0/4121324991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79700fbd70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f796c004660 tx=0x7f796c009b30).stop 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- 172.21.15.84:0/4121324991 shutdown_connections 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 --2- 172.21.15.84:0/4121324991 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79700fbd70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 --2- 172.21.15.84:0/4121324991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 0x7f79700f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- 172.21.15.84:0/4121324991 >> 172.21.15.84:0/4121324991 conn(0x7f79700f0e20 msgr2=0x7f79700f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- 172.21.15.84:0/4121324991 shutdown_connections 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- 172.21.15.84:0/4121324991 wait complete. 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 Processor -- start 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- start start 2024-01-18T07:34:00.386 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 0x7f7970107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7970107ba0 con 0x7f79700f9920 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.383+0000 7f7977d70700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7970107ce0 con 0x7f7970060870 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57756/0 (socket says 172.21.15.84:57756) 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 -- 172.21.15.84:0/3495531185 learned_addr learned my addr 172.21.15.84:0/3495531185 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f7975b0c700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 0x7f7970107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 -- 172.21.15.84:0/3495531185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 msgr2=0x7f7970107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 0x7f7970107100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 -- 172.21.15.84:0/3495531185 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c005040 con 0x7f79700f9920 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f797530b700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f796c009830 tx=0x7f796c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c018070 con 0x7f79700f9920 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7970108620 con 0x7f79700f9920 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f796c007ba0 con 0x7f79700f9920 2024-01-18T07:34:00.387 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f796c01de20 con 0x7f79700f9920 2024-01-18T07:34:00.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.384+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7970108b80 con 0x7f79700f9920 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.385+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7970045ad0 con 0x7f79700f9920 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.385+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f796c007d10 con 0x7f79700f9920 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.386+0000 7f79667fc700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 0x7f795c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.386+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f796c077730 con 0x7f79700f9920 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.386+0000 7f7975b0c700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 0x7f795c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:00.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.386+0000 7f7975b0c700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 0x7f795c05e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7960004c40 tx=0x7f796000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:00.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.390+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f796c04a730 con 0x7f79700f9920 2024-01-18T07:34:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:33:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:33:59.000211+0000 mgr.smithi084.sihbqg (mgr.14162) 490 : cluster [DBG] pgmap v458: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:00.540 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.539+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7970027a00 con 0x7f79700f9920 2024-01-18T07:34:00.541 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.540+0000 7f79667fc700 1 -- 172.21.15.84:0/3495531185 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10923 (secure 0 0 0) 0x7f796c04dd50 con 0x7f79700f9920 2024-01-18T07:34:00.541 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:00.541 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","created":"2024-01-18T07:18:42.511399+0000","modified":"2024-01-18T07:33:54.806303+0000","last_up_change":"2024-01-18T07:33:53.799733+0000","last_in_change":"2024-01-18T07:33:38.842493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-18T07:28:25.827946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2490ef18-5987-444d-80e1-a7ee0dd798a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6803","nonce":2976263962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6805","nonce":2976263962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6809","nonce":2976263962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6807","nonce":2976263962}]},"public_addr":"172.21.15.84:6803/2976263962","cluster_addr":"172.21.15.84:6805/2976263962","heartbeat_back_addr":"172.21.15.84:6809/2976263962","heartbeat_front_addr":"172.21.15.84:6807/2976263962","state":["exists","up"]},{"osd":1,"uuid":"b89774fa-ff6c-49dc-91d4-296a30dbb3a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6811","nonce":3205743070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6813","nonce":3205743070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6817","nonce":3205743070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6815","nonce":3205743070}]},"public_addr":"172.21.15.84:6811/3205743070","cluster_addr":"172.21.15.84:6813/3205743070","heartbeat_back_addr":"172.21.15.84:6817/3205743070","heartbeat_front_addr":"172.21.15.84:6815/3205743070","state":["exists","up"]},{"osd":2,"uuid":"0543c962-6df6-4a1b-8c7a-1335c7c5f501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6819","nonce":3571639289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6821","nonce":3571639289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6825","nonce":3571639289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6823","nonce":3571639289}]},"public_addr":"172.21.15.84:6819/3571639289","cluster_addr":"172.21.15.84:6821/3571639289","heartbeat_back_addr":"172.21.15.84:6825/3571639289","heartbeat_front_addr":"172.21.15.84:6823/3571639289","state":["exists","up"]},{"osd":3,"uuid":"cbac515c-d096-47c9-ada0-a838b722770e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6801","nonce":3853148779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6803","nonce":3853148779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6807","nonce":3853148779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6805","nonce":3853148779}]},"public_addr":"172.21.15.106:6801/3853148779","cluster_addr":"172.21.15.106:6803/3853148779","heartbeat_back_addr":"172.21.15.106:6807/3853148779","heartbeat_front_addr":"172.21.15.106:6805/3853148779","state":["exists","up"]},{"osd":4,"uuid":"def49df5-fdbc-421f-9787-8017f3349879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6809","nonce":4131077435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6811","nonce":4131077435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6815","nonce":4131077435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6813","nonce":4131077435}]},"public_addr":"172.21.15.106:6809/4131077435","cluster_addr":"172.21.15.106:6811/4131077435","heartbeat_back_addr":"172.21.15.106:6815/4131077435","heartbeat_front_addr":"172.21.15.106:6813/4131077435","state":["exists","up"]},{"osd":5,"uuid":"9c59b868-d177-4948-9e86-cb37faa7ce12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6817","nonce":408419143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6819","nonce":408419143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6823","nonce":408419143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6821","nonce":408419143}]},"public_addr":"172.21.15.106:6817/408419143","cluster_addr":"172.21.15.106:6819/408419143","heartbeat_back_addr":"172.21.15.106:6823/408419143","heartbeat_front_addr":"172.21.15.106:6821/408419143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:13.753014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:26.797564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:40.886392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:28:30.539270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:31:41.334918+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:33:51.898063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:6801/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:6800/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/4248541276":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/663309230":"2024-01-19T07:19:22.910646+0000","172.21.15.84:6801/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3646087192":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3147600455":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/2489337289":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6801/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/1904953819":"2024-01-19T07:19:00.115732+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 msgr2=0x7f795c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 0x7f795c05e760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7960004c40 tx=0x7f796000b040).stop 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 msgr2=0x7f79701075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f796c009830 tx=0x7f796c009b30).stop 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 shutdown_connections 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f795c05c2a0 0x7f795c05e760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7970060870 0x7f7970107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 --2- 172.21.15.84:0/3495531185 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f79700f9920 0x7f79701075f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 >> 172.21.15.84:0/3495531185 conn(0x7f79700f0e20 msgr2=0x7f79700f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 shutdown_connections 2024-01-18T07:34:00.543 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:00.541+0000 7f7977d70700 1 -- 172.21.15.84:0/3495531185 wait complete. 2024-01-18T07:34:00.934 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-01-18T07:28:25.827946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}] 2024-01-18T07:34:00.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-01-18T07:34:01.195 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:01.313 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:00.541297+0000 mon.smithi084 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.84:0/3495531185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:00.541297+0000 mon.smithi084 (mon.0) 491 : audit [DBG] from='client.? 172.21.15.84:0/3495531185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/34432010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 msgr2=0x7fcc740f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/34432010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc740f6c10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fcc70004660 tx=0x7fcc70009b30).stop 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/34432010 shutdown_connections 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/34432010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc740f6c10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/34432010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 0x7fcc740f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/34432010 >> 172.21.15.84:0/34432010 conn(0x7fcc740f0c70 msgr2=0x7fcc740f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/34432010 shutdown_connections 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.245+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/34432010 wait complete. 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 Processor -- start 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 -- start start 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 0x7fcc7410bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:02.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74060810 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc7aa3b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc74060950 con 0x7fcc740f5510 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc79238700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc79238700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57780/0 (socket says 172.21.15.84:57780) 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc79238700 1 -- 172.21.15.84:0/1070868410 learned_addr learned my addr 172.21.15.84:0/1070868410 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.246+0000 7fcc79a39700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 0x7fcc7410bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc79238700 1 -- 172.21.15.84:0/1070868410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 msgr2=0x7fcc7410bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc79238700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 0x7fcc7410bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc79238700 1 -- 172.21.15.84:0/1070868410 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc70005040 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc79238700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fcc70009830 tx=0x7fcc70009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc70018070 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc70013410 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7410e9e0 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc7001c620 con 0x7fcc740f67b0 2024-01-18T07:34:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.247+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7410edb0 con 0x7fcc740f67b0 2024-01-18T07:34:02.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.248+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc74045ad0 con 0x7fcc740f67b0 2024-01-18T07:34:02.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.248+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcc7001c8e0 con 0x7fcc740f67b0 2024-01-18T07:34:02.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.248+0000 7fcc6a7fc700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 0x7fcc6005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:02.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.249+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcc70077a80 con 0x7fcc740f67b0 2024-01-18T07:34:02.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.249+0000 7fcc79a39700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 0x7fcc6005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:02.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.249+0000 7fcc79a39700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 0x7fcc6005e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcc740605a0 tx=0x7fcc64009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:02.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.252+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc7004aa20 con 0x7fcc740f67b0 2024-01-18T07:34:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:01.000564+0000 mgr.smithi084.sihbqg (mgr.14162) 491 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:02.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.402+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fcc74027a00 con 0x7fcc740f67b0 2024-01-18T07:34:02.404 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.403+0000 7fcc6a7fc700 1 -- 172.21.15.84:0/1070868410 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v37) v1 ==== 110+0+10 (secure 0 0 0) 0x7fcc7004e040 con 0x7fcc740f67b0 2024-01-18T07:34:02.404 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 msgr2=0x7fcc6005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 0x7fcc6005e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcc740605a0 tx=0x7fcc64009040).stop 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 msgr2=0x7fcc7410e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fcc70009830 tx=0x7fcc70009b30).stop 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 shutdown_connections 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcc6005c2b0 0x7fcc6005e770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc740f5510 0x7fcc7410bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 --2- 172.21.15.84:0/1070868410 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcc740f67b0 0x7fcc7410e4f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:02.406 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.404+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 >> 172.21.15.84:0/1070868410 conn(0x7fcc740f0c70 msgr2=0x7fcc740fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:02.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.405+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 shutdown_connections 2024-01-18T07:34:02.407 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:02.405+0000 7fcc7aa3b700 1 -- 172.21.15.84:0/1070868410 wait complete. 2024-01-18T07:34:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:01.000564+0000 mgr.smithi084.sihbqg (mgr.14162) 491 : cluster [DBG] pgmap v459: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:02.744 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-18T07:34:02.744 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-01-18T07:34:02.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph mgr dump --format=json 2024-01-18T07:34:02.966 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:03.311 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:02.404889+0000 mon.smithi084 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.84:0/1070868410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-01-18T07:34:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:02.404889+0000 mon.smithi084 (mon.0) 492 : audit [DBG] from='client.? 172.21.15.84:0/1070868410' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-01-18T07:34:04.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 -- 172.21.15.84:0/1574583680 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed200f9980 msgr2=0x7fed200fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:04.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 --2- 172.21.15.84:0/1574583680 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed200f9980 0x7fed200fbd70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fed1c004660 tx=0x7fed1c009b30).stop 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 -- 172.21.15.84:0/1574583680 shutdown_connections 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 --2- 172.21.15.84:0/1574583680 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed200f9980 0x7fed200fbd70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 --2- 172.21.15.84:0/1574583680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20060810 0x7fed200f9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 -- 172.21.15.84:0/1574583680 >> 172.21.15.84:0/1574583680 conn(0x7fed200f0e20 msgr2=0x7fed200f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 -- 172.21.15.84:0/1574583680 shutdown_connections 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.025+0000 7fed272cc700 1 -- 172.21.15.84:0/1574583680 wait complete. 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 Processor -- start 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 -- start start 2024-01-18T07:34:04.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20107700 0x7fed201085f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20107bd0 con 0x7fed20060810 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed20107d10 con 0x7fed20107700 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57792/0 (socket says 172.21.15.84:57792) 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 -- 172.21.15.84:0/1251177946 learned_addr learned my addr 172.21.15.84:0/1251177946 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed24867700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20107700 0x7fed201085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 -- 172.21.15.84:0/1251177946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20107700 msgr2=0x7fed201085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20107700 0x7fed201085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 -- 172.21.15.84:0/1251177946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c005040 con 0x7fed20060810 2024-01-18T07:34:04.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed25068700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fed1000c530 tx=0x7fed10007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1000f650 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed1000fc60 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.026+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed20108ba0 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.027+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed10012c80 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.027+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed20109160 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.028+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed20045ad0 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.028+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fed10010070 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.028+0000 7fed15ffb700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 0x7fed0c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.028+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fed100758c0 con 0x7fed20060810 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.028+0000 7fed24867700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 0x7fed0c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:04.033 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.029+0000 7fed24867700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 0x7fed0c05e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fed1c000c00 tx=0x7fed1c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:04.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.032+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed100452b0 con 0x7fed20060810 2024-01-18T07:34:04.220 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.218+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fed20109380 con 0x7fed20060810 2024-01-18T07:34:04.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.221+0000 7fed15ffb700 1 -- 172.21.15.84:0/1251177946 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v18) v1 ==== 74+0+143500 (secure 0 0 0) 0x7fed100430a0 con 0x7fed20060810 2024-01-18T07:34:04.226 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:04.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 msgr2=0x7fed0c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:04.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 0x7fed0c05e760 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fed1c000c00 tx=0x7fed1c009b30).stop 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 msgr2=0x7fed20107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fed1000c530 tx=0x7fed10007e20).stop 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 shutdown_connections 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fed20060810 0x7fed20107210 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fed0c05c2a0 0x7fed0c05e760 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 --2- 172.21.15.84:0/1251177946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fed20107700 0x7fed201085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:04.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 >> 172.21.15.84:0/1251177946 conn(0x7fed200f0e20 msgr2=0x7fed200f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:04.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 shutdown_connections 2024-01-18T07:34:04.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:04.223+0000 7fed272cc700 1 -- 172.21.15.84:0/1251177946 wait complete. 2024-01-18T07:34:04.253 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:03.000975+0000 mgr.smithi084.sihbqg (mgr.14162) 492 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:04.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:03.000975+0000 mgr.smithi084.sihbqg (mgr.14162) 492 : cluster [DBG] pgmap v460: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:04.650 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":18,"active_gid":14162,"active_name":"smithi084.sihbqg","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":2258764086},{"type":"v1","addr":"172.21.15.84:6801","nonce":2258764086}]},"active_addr":"172.21.15.84:6801/2258764086","active_change":"2024-01-18T07:19:34.840995+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":14334,"name":"smithi106.yhaxbc","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://smithi084.front.sepia.ceph.com:8443/","prometheus":"http://smithi084.front.sepia.ceph.com:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":4,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":2551384162}]},{"addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":3770379640}]}]}} 2024-01-18T07:34:04.655 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-01-18T07:34:04.655 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-01-18T07:34:04.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-18T07:34:04.877 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:05.364 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:04.220936+0000 mon.smithi084 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.84:0/1251177946' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-18T07:34:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:04.220936+0000 mon.smithi084 (mon.0) 493 : audit [DBG] from='client.? 172.21.15.84:0/1251177946' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 -- 172.21.15.84:0/4210414911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 msgr2=0x7fb1980f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 --2- 172.21.15.84:0/4210414911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f93a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fb188004660 tx=0x7fb188009b30).stop 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 -- 172.21.15.84:0/4210414911 shutdown_connections 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 --2- 172.21.15.84:0/4210414911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1980f9920 0x7fb1980fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 --2- 172.21.15.84:0/4210414911 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f93a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 -- 172.21.15.84:0/4210414911 >> 172.21.15.84:0/4210414911 conn(0x7fb1980f0e20 msgr2=0x7fb1980f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 -- 172.21.15.84:0/4210414911 shutdown_connections 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 -- 172.21.15.84:0/4210414911 wait complete. 2024-01-18T07:34:05.877 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.874+0000 7fb19df60700 1 Processor -- start 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 -- start start 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1980f9920 0x7fb1980f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1980f5890 con 0x7fb198060870 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1980f59d0 con 0x7fb1980f9920 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:57802/0 (socket says 172.21.15.84:57802) 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 -- 172.21.15.84:0/1638752675 learned_addr learned my addr 172.21.15.84:0/1638752675 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 -- 172.21.15.84:0/1638752675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1980f9920 msgr2=0x7fb1980f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1980f9920 0x7fb1980f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:05.878 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 -- 172.21.15.84:0/1638752675 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb188005040 con 0x7fb198060870 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb1977fe700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fb188009830 tx=0x7fb188009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb188018070 con 0x7fb198060870 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1980f5c50 con 0x7fb198060870 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.875+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb188013410 con 0x7fb198060870 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.876+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb18801c620 con 0x7fb198060870 2024-01-18T07:34:05.879 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.876+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19810f0d0 con 0x7fb198060870 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.876+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb198045ad0 con 0x7fb198060870 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.877+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb18801c8e0 con 0x7fb198060870 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.877+0000 7fb17ffff700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 0x7fb18005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.877+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb188077a60 con 0x7fb198060870 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.877+0000 7fb196ffd700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 0x7fb18005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:05.880 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.878+0000 7fb196ffd700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 0x7fb18005e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb198060640 tx=0x7fb18c006b50).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:05.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:05.881+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb18804aa00 con 0x7fb198060870 2024-01-18T07:34:06.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.033+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb198027a00 con 0x7fb198060870 2024-01-18T07:34:06.035 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.034+0000 7fb17ffff700 1 -- 172.21.15.84:0/1638752675 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10923 (secure 0 0 0) 0x7fb18804e020 con 0x7fb198060870 2024-01-18T07:34:06.035 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:06.035 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","created":"2024-01-18T07:18:42.511399+0000","modified":"2024-01-18T07:33:54.806303+0000","last_up_change":"2024-01-18T07:33:53.799733+0000","last_in_change":"2024-01-18T07:33:38.842493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-18T07:28:25.827946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2490ef18-5987-444d-80e1-a7ee0dd798a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6803","nonce":2976263962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6805","nonce":2976263962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6809","nonce":2976263962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6807","nonce":2976263962}]},"public_addr":"172.21.15.84:6803/2976263962","cluster_addr":"172.21.15.84:6805/2976263962","heartbeat_back_addr":"172.21.15.84:6809/2976263962","heartbeat_front_addr":"172.21.15.84:6807/2976263962","state":["exists","up"]},{"osd":1,"uuid":"b89774fa-ff6c-49dc-91d4-296a30dbb3a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6811","nonce":3205743070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6813","nonce":3205743070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6817","nonce":3205743070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6815","nonce":3205743070}]},"public_addr":"172.21.15.84:6811/3205743070","cluster_addr":"172.21.15.84:6813/3205743070","heartbeat_back_addr":"172.21.15.84:6817/3205743070","heartbeat_front_addr":"172.21.15.84:6815/3205743070","state":["exists","up"]},{"osd":2,"uuid":"0543c962-6df6-4a1b-8c7a-1335c7c5f501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6819","nonce":3571639289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6821","nonce":3571639289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6825","nonce":3571639289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6823","nonce":3571639289}]},"public_addr":"172.21.15.84:6819/3571639289","cluster_addr":"172.21.15.84:6821/3571639289","heartbeat_back_addr":"172.21.15.84:6825/3571639289","heartbeat_front_addr":"172.21.15.84:6823/3571639289","state":["exists","up"]},{"osd":3,"uuid":"cbac515c-d096-47c9-ada0-a838b722770e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6801","nonce":3853148779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6803","nonce":3853148779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6807","nonce":3853148779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6805","nonce":3853148779}]},"public_addr":"172.21.15.106:6801/3853148779","cluster_addr":"172.21.15.106:6803/3853148779","heartbeat_back_addr":"172.21.15.106:6807/3853148779","heartbeat_front_addr":"172.21.15.106:6805/3853148779","state":["exists","up"]},{"osd":4,"uuid":"def49df5-fdbc-421f-9787-8017f3349879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6809","nonce":4131077435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6811","nonce":4131077435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6815","nonce":4131077435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6813","nonce":4131077435}]},"public_addr":"172.21.15.106:6809/4131077435","cluster_addr":"172.21.15.106:6811/4131077435","heartbeat_back_addr":"172.21.15.106:6815/4131077435","heartbeat_front_addr":"172.21.15.106:6813/4131077435","state":["exists","up"]},{"osd":5,"uuid":"9c59b868-d177-4948-9e86-cb37faa7ce12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6817","nonce":408419143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6819","nonce":408419143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6823","nonce":408419143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6821","nonce":408419143}]},"public_addr":"172.21.15.106:6817/408419143","cluster_addr":"172.21.15.106:6819/408419143","heartbeat_back_addr":"172.21.15.106:6823/408419143","heartbeat_front_addr":"172.21.15.106:6821/408419143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:13.753014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:26.797564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:40.886392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:28:30.539270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:31:41.334918+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:33:51.898063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:6801/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:6800/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/4248541276":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/663309230":"2024-01-19T07:19:22.910646+0000","172.21.15.84:6801/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3646087192":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3147600455":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/2489337289":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6801/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/1904953819":"2024-01-19T07:19:00.115732+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 msgr2=0x7fb18005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 0x7fb18005e760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb198060640 tx=0x7fb18c006b50).stop 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 msgr2=0x7fb1980f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fb188009830 tx=0x7fb188009b30).stop 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 shutdown_connections 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb198060870 0x7fb1980f6e20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb18005c2a0 0x7fb18005e760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:06.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 --2- 172.21.15.84:0/1638752675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1980f9920 0x7fb1980f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:06.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.035+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 >> 172.21.15.84:0/1638752675 conn(0x7fb1980f0e20 msgr2=0x7fb1980f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:06.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.036+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 shutdown_connections 2024-01-18T07:34:06.038 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:06.036+0000 7fb19df60700 1 -- 172.21.15.84:0/1638752675 wait complete. 2024-01-18T07:34:06.263 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:05.001302+0000 mgr.smithi084.sihbqg (mgr.14162) 493 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:06.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:05.001302+0000 mgr.smithi084.sihbqg (mgr.14162) 493 : cluster [DBG] pgmap v461: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:06.483 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-01-18T07:34:06.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd dump --format=json 2024-01-18T07:34:06.703 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:06.035493+0000 mon.smithi084 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.84:0/1638752675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:07.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:07.001663+0000 mgr.smithi084.sihbqg (mgr.14162) 494 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:06.035493+0000 mon.smithi084 (mon.0) 494 : audit [DBG] from='client.? 172.21.15.84:0/1638752675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:07.001663+0000 mgr.smithi084.sihbqg (mgr.14162) 494 : cluster [DBG] pgmap v462: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- 172.21.15.84:0/3283567251 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 msgr2=0x7f64ec0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 --2- 172.21.15.84:0/3283567251 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec0f5ae0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f64d4004660 tx=0x7f64d4009b30).stop 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- 172.21.15.84:0/3283567251 shutdown_connections 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 --2- 172.21.15.84:0/3283567251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 --2- 172.21.15.84:0/3283567251 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec0f5ae0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- 172.21.15.84:0/3283567251 >> 172.21.15.84:0/3283567251 conn(0x7f64ec0f0e20 msgr2=0x7f64ec0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- 172.21.15.84:0/3283567251 shutdown_connections 2024-01-18T07:34:07.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- 172.21.15.84:0/3283567251 wait complete. 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 Processor -- start 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- start start 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec1075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec107b90 con 0x7f64ec0f56c0 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.728+0000 7f64f0997700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ec107cd0 con 0x7f64ec0f6960 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32798/0 (socket says 172.21.15.84:32798) 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 -- 172.21.15.84:0/1280054801 learned_addr learned my addr 172.21.15.84:0/1280054801 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64e9d9b700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec1075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 -- 172.21.15.84:0/1280054801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 msgr2=0x7f64ec1075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec1075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 -- 172.21.15.84:0/1280054801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4005040 con 0x7f64ec0f56c0 2024-01-18T07:34:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64e9d9b700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec1075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64ea59c700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f64d4009f50 tx=0x7f64d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d4018070 con 0x7f64ec0f56c0 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64d4013410 con 0x7f64ec0f56c0 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d401c620 con 0x7f64ec0f56c0 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ec108610 con 0x7f64ec0f56c0 2024-01-18T07:34:07.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.729+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ec108b70 con 0x7f64ec0f56c0 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.730+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ec045ad0 con 0x7f64ec0f56c0 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.730+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f64d401d420 con 0x7f64ec0f56c0 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.731+0000 7f64e2ffd700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 0x7f64d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.731+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f64d40779f0 con 0x7f64ec0f56c0 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.731+0000 7f64e9d9b700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 0x7f64d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:07.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.731+0000 7f64e9d9b700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 0x7f64d805e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f64dc004a70 tx=0x7f64dc009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:07.736 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.735+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64d4079e60 con 0x7f64ec0f56c0 2024-01-18T07:34:07.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.883+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64ec0fde50 con 0x7f64ec0f56c0 2024-01-18T07:34:07.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.884+0000 7f64e2ffd700 1 -- 172.21.15.84:0/1280054801 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+10923 (secure 0 0 0) 0x7f64d404a9f0 con 0x7f64ec0f56c0 2024-01-18T07:34:07.886 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:07.886 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":37,"fsid":"bf9448d2-b5d1-11ee-95ae-87774f69a715","created":"2024-01-18T07:18:42.511399+0000","modified":"2024-01-18T07:33:54.806303+0000","last_up_change":"2024-01-18T07:33:53.799733+0000","last_in_change":"2024-01-18T07:33:38.842493+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-18T07:28:25.827946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2490ef18-5987-444d-80e1-a7ee0dd798a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6803","nonce":2976263962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6805","nonce":2976263962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6809","nonce":2976263962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":2976263962},{"type":"v1","addr":"172.21.15.84:6807","nonce":2976263962}]},"public_addr":"172.21.15.84:6803/2976263962","cluster_addr":"172.21.15.84:6805/2976263962","heartbeat_back_addr":"172.21.15.84:6809/2976263962","heartbeat_front_addr":"172.21.15.84:6807/2976263962","state":["exists","up"]},{"osd":1,"uuid":"b89774fa-ff6c-49dc-91d4-296a30dbb3a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6811","nonce":3205743070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6813","nonce":3205743070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6817","nonce":3205743070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3205743070},{"type":"v1","addr":"172.21.15.84:6815","nonce":3205743070}]},"public_addr":"172.21.15.84:6811/3205743070","cluster_addr":"172.21.15.84:6813/3205743070","heartbeat_back_addr":"172.21.15.84:6817/3205743070","heartbeat_front_addr":"172.21.15.84:6815/3205743070","state":["exists","up"]},{"osd":2,"uuid":"0543c962-6df6-4a1b-8c7a-1335c7c5f501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6818","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6819","nonce":3571639289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6820","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6821","nonce":3571639289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6824","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6825","nonce":3571639289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6822","nonce":3571639289},{"type":"v1","addr":"172.21.15.84:6823","nonce":3571639289}]},"public_addr":"172.21.15.84:6819/3571639289","cluster_addr":"172.21.15.84:6821/3571639289","heartbeat_back_addr":"172.21.15.84:6825/3571639289","heartbeat_front_addr":"172.21.15.84:6823/3571639289","state":["exists","up"]},{"osd":3,"uuid":"cbac515c-d096-47c9-ada0-a838b722770e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6801","nonce":3853148779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6803","nonce":3853148779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6807","nonce":3853148779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3853148779},{"type":"v1","addr":"172.21.15.106:6805","nonce":3853148779}]},"public_addr":"172.21.15.106:6801/3853148779","cluster_addr":"172.21.15.106:6803/3853148779","heartbeat_back_addr":"172.21.15.106:6807/3853148779","heartbeat_front_addr":"172.21.15.106:6805/3853148779","state":["exists","up"]},{"osd":4,"uuid":"def49df5-fdbc-421f-9787-8017f3349879","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6809","nonce":4131077435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6811","nonce":4131077435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6815","nonce":4131077435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":4131077435},{"type":"v1","addr":"172.21.15.106:6813","nonce":4131077435}]},"public_addr":"172.21.15.106:6809/4131077435","cluster_addr":"172.21.15.106:6811/4131077435","heartbeat_back_addr":"172.21.15.106:6815/4131077435","heartbeat_front_addr":"172.21.15.106:6813/4131077435","state":["exists","up"]},{"osd":5,"uuid":"9c59b868-d177-4948-9e86-cb37faa7ce12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6817","nonce":408419143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6819","nonce":408419143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6823","nonce":408419143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":408419143},{"type":"v1","addr":"172.21.15.106:6821","nonce":408419143}]},"public_addr":"172.21.15.106:6817/408419143","cluster_addr":"172.21.15.106:6819/408419143","heartbeat_back_addr":"172.21.15.106:6823/408419143","heartbeat_front_addr":"172.21.15.106:6821/408419143","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:13.753014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:26.797564+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:24:40.886392+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:28:30.539270+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:31:41.334918+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2024-01-18T07:33:51.898063+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:6801/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:6800/2270362783":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/4248541276":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/663309230":"2024-01-19T07:19:22.910646+0000","172.21.15.84:6801/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3646087192":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/3147600455":"2024-01-19T07:19:34.840638+0000","172.21.15.84:0/2489337289":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6801/2132670921":"2024-01-19T07:19:00.115732+0000","172.21.15.84:6800/2693695158":"2024-01-19T07:19:22.910646+0000","172.21.15.84:0/1904953819":"2024-01-19T07:19:00.115732+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T07:34:07.887 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 msgr2=0x7f64d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 0x7f64d805e760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f64dc004a70 tx=0x7f64dc009040).stop 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 msgr2=0x7f64ec1070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f64d4009f50 tx=0x7f64d4009b30).stop 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 shutdown_connections 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64ec0f56c0 0x7f64ec1070f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.888 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64d805c2a0 0x7f64d805e760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 --2- 172.21.15.84:0/1280054801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64ec0f6960 0x7f64ec1075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:07.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 >> 172.21.15.84:0/1280054801 conn(0x7f64ec0f0e20 msgr2=0x7f64ec0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:07.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 shutdown_connections 2024-01-18T07:34:07.889 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:07.886+0000 7f64f0997700 1 -- 172.21.15.84:0/1280054801 wait complete. 2024-01-18T07:34:08.216 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:07.885796+0000 mon.smithi084 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.84:0/1280054801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:08.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-01-18T07:34:08.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:07.885796+0000 mon.smithi084 (mon.0) 495 : audit [DBG] from='client.? 172.21.15.84:0/1280054801' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-18T07:34:08.469 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:09.290 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:09.002144+0000 mgr.smithi084.sihbqg (mgr.14162) 495 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:09.002144+0000 mgr.smithi084.sihbqg (mgr.14162) 495 : cluster [DBG] pgmap v463: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 -- 172.21.15.84:0/4132002158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f5510 msgr2=0x7f95a00f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/4132002158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f5510 0x7f95a00f5930 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f9590004660 tx=0x7f9590009b30).stop 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 -- 172.21.15.84:0/4132002158 shutdown_connections 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/4132002158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f67b0 0x7f95a00f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/4132002158 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f5510 0x7f95a00f5930 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 -- 172.21.15.84:0/4132002158 >> 172.21.15.84:0/4132002158 conn(0x7f95a00f0c70 msgr2=0x7f95a00f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 -- 172.21.15.84:0/4132002158 shutdown_connections 2024-01-18T07:34:09.475 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.473+0000 7f95a6b8b700 1 -- 172.21.15.84:0/4132002158 wait complete. 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 Processor -- start 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 -- start start 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f5510 0x7f95a0105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0106940 con 0x7f95a00f67b0 2024-01-18T07:34:09.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a6b8b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0107140 con 0x7f95a00f5510 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32820/0 (socket says 172.21.15.84:32820) 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 -- 172.21.15.84:0/1801521295 learned_addr learned my addr 172.21.15.84:0/1801521295 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5b89700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f5510 0x7f95a0105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 -- 172.21.15.84:0/1801521295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f5510 msgr2=0x7f95a0105ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f5510 0x7f95a0105ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.474+0000 7f95a5388700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590005040 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95a5388700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f959c0026d0 tx=0x7f959c009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f959c0079c0 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f959c00f040 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a01073a0 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f959c0148b0 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a0107930 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.475+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95a005db80 con 0x7f95a00f67b0 2024-01-18T07:34:09.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f959c007b20 con 0x7f95a00f67b0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 0x7f958c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f959c075b00 con 0x7f95a00f67b0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95a5b89700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 0x7f958c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 0x7f958c0644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f958c064b20 con 0x7f958c062040 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.476+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f959c07a220 con 0x7f95a00f67b0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.477+0000 7f95a638a700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 0x7f958c0644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.477+0000 7f95a5b89700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 0x7f958c05e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f9590009f50 tx=0x7f9590009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.477+0000 7f95a638a700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 0x7f958c0644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:09.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.477+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== osd.0 v2:172.21.15.84:6802/2976263962 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f958c064b20 con 0x7f958c062040 2024-01-18T07:34:09.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 --> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95a0107e70 con 0x7f958c062040 2024-01-18T07:34:09.493 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95967fc700 1 -- 172.21.15.84:0/1801521295 <== osd.0 v2:172.21.15.84:6802/2976263962 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f95a0107e70 con 0x7f958c062040 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 msgr2=0x7f958c0644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 0x7f958c0644a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 msgr2=0x7f958c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 0x7f958c05e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f9590009f50 tx=0x7f9590009b30).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 msgr2=0x7f95a0106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f959c0026d0 tx=0x7f959c009de0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 shutdown_connections 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6802/2976263962,v1:172.21.15.84:6803/2976263962] conn(0x7f958c062040 0x7f958c0644a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f958c05c2b0 0x7f958c05e770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95a00f5510 0x7f95a0105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 --2- 172.21.15.84:0/1801521295 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f95a00f67b0 0x7f95a0106390 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:09.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 >> 172.21.15.84:0/1801521295 conn(0x7f95a00f0c70 msgr2=0x7f95a00fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:09.495 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 shutdown_connections 2024-01-18T07:34:09.495 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:09.493+0000 7f95a6b8b700 1 -- 172.21.15.84:0/1801521295 wait complete. 2024-01-18T07:34:09.925 INFO:teuthology.orchestra.run.smithi084.stdout:38654705784 2024-01-18T07:34:09.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-01-18T07:34:10.145 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:11.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- 172.21.15.84:0/4279801436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 msgr2=0x7f53f40f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:11.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 --2- 172.21.15.84:0/4279801436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f40f7ec0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f53e4004660 tx=0x7f53e4009b30).stop 2024-01-18T07:34:11.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- 172.21.15.84:0/4279801436 shutdown_connections 2024-01-18T07:34:11.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 --2- 172.21.15.84:0/4279801436 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f40f7ec0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 --2- 172.21.15.84:0/4279801436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 0x7f53f40f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- 172.21.15.84:0/4279801436 >> 172.21.15.84:0/4279801436 conn(0x7f53f40f1f20 msgr2=0x7f53f40f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- 172.21.15.84:0/4279801436 shutdown_connections 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- 172.21.15.84:0/4279801436 wait complete. 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 Processor -- start 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.134+0000 7f53f88bc700 1 -- start start 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 0x7f53f41070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f4107be0 con 0x7f53f40f7a60 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f4108490 con 0x7f53f40f67c0 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32840/0 (socket says 172.21.15.84:32840) 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 -- 172.21.15.84:0/597242580 learned_addr learned my addr 172.21.15.84:0/597242580 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f259c700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 0x7f53f41070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:11.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 -- 172.21.15.84:0/597242580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 msgr2=0x7f53f41070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 0x7f53f41070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e4005040 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f1d9b700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f53e4009830 tx=0x7f53e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e4018070 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e4013410 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f4108710 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e401c620 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.135+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f4108c70 con 0x7f53f40f7a60 2024-01-18T07:34:11.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.136+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53f405dd30 con 0x7f53f40f7a60 2024-01-18T07:34:11.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.136+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f53e401c890 con 0x7f53f40f7a60 2024-01-18T07:34:11.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53e2ffd700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 0x7f53dc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:11.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f53e4077eb0 con 0x7f53f40f7a60 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53f259c700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 0x7f53dc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53e2ffd700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 0x7f53dc064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53dc064b10 con 0x7f53dc062030 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e40782b0 con 0x7f53f40f7a60 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53f2d9d700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 0x7f53dc064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53f259c700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 0x7f53dc05e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f53f40f78c0 tx=0x7f53e8008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.137+0000 7f53f2d9d700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 0x7f53dc064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:11.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.138+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== osd.1 v2:172.21.15.84:6810/3205743070 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f53dc064b10 con 0x7f53dc062030 2024-01-18T07:34:11.152 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.152+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 --> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f53f4109120 con 0x7f53dc062030 2024-01-18T07:34:11.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.152+0000 7f53e2ffd700 1 -- 172.21.15.84:0/597242580 <== osd.1 v2:172.21.15.84:6810/3205743070 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f53f4109120 con 0x7f53dc062030 2024-01-18T07:34:11.153 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 msgr2=0x7f53dc064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 0x7f53dc064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 msgr2=0x7f53dc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 0x7f53dc05e760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f53f40f78c0 tx=0x7f53e8008040).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 msgr2=0x7f53f41075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f53e4009830 tx=0x7f53e4009b30).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 shutdown_connections 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f53dc05c2a0 0x7f53dc05e760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53f40f67c0 0x7f53f41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:6810/3205743070,v1:172.21.15.84:6811/3205743070] conn(0x7f53dc062030 0x7f53dc064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 --2- 172.21.15.84:0/597242580 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53f40f7a60 0x7f53f41075a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 >> 172.21.15.84:0/597242580 conn(0x7f53f40f1f20 msgr2=0x7f53f40fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 shutdown_connections 2024-01-18T07:34:11.154 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:11.153+0000 7f53f88bc700 1 -- 172.21.15.84:0/597242580 wait complete. 2024-01-18T07:34:11.329 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:11.002397+0000 mgr.smithi084.sihbqg (mgr.14162) 496 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:11.002397+0000 mgr.smithi084.sihbqg (mgr.14162) 496 : cluster [DBG] pgmap v464: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:11.541 INFO:teuthology.orchestra.run.smithi084.stdout:55834574967 2024-01-18T07:34:11.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-01-18T07:34:11.762 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 -- 172.21.15.84:0/1547287416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f7a70 msgr2=0x7f7c4c0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 --2- 172.21.15.84:0/1547287416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c0f7ed0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f7c3c004660 tx=0x7f7c3c009b30).stop 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 -- 172.21.15.84:0/1547287416 shutdown_connections 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 --2- 172.21.15.84:0/1547287416 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c0f7ed0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 --2- 172.21.15.84:0/1547287416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 -- 172.21.15.84:0/1547287416 >> 172.21.15.84:0/1547287416 conn(0x7f7c4c0f1f50 msgr2=0x7f7c4c0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 -- 172.21.15.84:0/1547287416 shutdown_connections 2024-01-18T07:34:12.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.707+0000 7f7c5119b700 1 -- 172.21.15.84:0/1547287416 wait complete. 2024-01-18T07:34:12.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 Processor -- start 2024-01-18T07:34:12.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 -- start start 2024-01-18T07:34:12.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:12.710 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c107b60 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c5119b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c107ca0 con 0x7f7c4c0f7a70 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c4ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c4ad9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32856/0 (socket says 172.21.15.84:32856) 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c4ad9d700 1 -- 172.21.15.84:0/3639023820 learned_addr learned my addr 172.21.15.84:0/3639023820 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.708+0000 7f7c4a59c700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c4ad9d700 1 -- 172.21.15.84:0/3639023820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f7a70 msgr2=0x7f7c4c1075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c4ad9d700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c1075b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c4ad9d700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3c005040 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c4ad9d700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f7c40002700 tx=0x7f7c40009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c400079c0 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c4000f040 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.711 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c108640 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c40014860 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.709+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c108c00 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.710+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7c4c05dd10 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.712 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.710+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7c4001b060 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.710+0000 7f7c337fe700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 0x7f7c3405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7c40075960 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c4a59c700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 0x7f7c3405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c337fe700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 0x7f7c34064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c34064b10 con 0x7f7c34062030 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c4007a220 con 0x7f7c4c0f67d0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c4b59e700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 0x7f7c34064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c4a59c700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 0x7f7c3405e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7c3c009f50 tx=0x7f7c3c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:12.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.711+0000 7f7c4b59e700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 0x7f7c34064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:12.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.712+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== osd.2 v2:172.21.15.84:6818/3571639289 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f7c34064b10 con 0x7f7c34062030 2024-01-18T07:34:12.726 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 --> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7c4c109190 con 0x7f7c34062030 2024-01-18T07:34:12.726 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c337fe700 1 -- 172.21.15.84:0/3639023820 <== osd.2 v2:172.21.15.84:6818/3571639289 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7c4c109190 con 0x7f7c34062030 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 msgr2=0x7f7c34064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 0x7f7c34064490 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 msgr2=0x7f7c3405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 0x7f7c3405e760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7c3c009f50 tx=0x7f7c3c009b30).stop 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 msgr2=0x7f7c4c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f7c40002700 tx=0x7f7c40009d80).stop 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 shutdown_connections 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6818/3571639289,v1:172.21.15.84:6819/3571639289] conn(0x7f7c34062030 0x7f7c34064490 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7c4c0f67d0 0x7f7c4c1070c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7c3405c2a0 0x7f7c3405e760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 --2- 172.21.15.84:0/3639023820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7c4c0f7a70 0x7f7c4c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:12.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 >> 172.21.15.84:0/3639023820 conn(0x7f7c4c0f1f50 msgr2=0x7f7c4c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:12.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 shutdown_connections 2024-01-18T07:34:12.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:12.726+0000 7f7c5119b700 1 -- 172.21.15.84:0/3639023820 wait complete. 2024-01-18T07:34:13.124 INFO:teuthology.orchestra.run.smithi084.stdout:73014444148 2024-01-18T07:34:13.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-01-18T07:34:13.343 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:13.002704+0000 mgr.smithi084.sihbqg (mgr.14162) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:13.002704+0000 mgr.smithi084.sihbqg (mgr.14162) 497 : cluster [DBG] pgmap v465: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:14.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 -- 172.21.15.84:0/3196481118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f6960 msgr2=0x7fcdc80f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3196481118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f6960 0x7fcdc80f6dc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fcdc4004660 tx=0x7fcdc4009b30).stop 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 -- 172.21.15.84:0/3196481118 shutdown_connections 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3196481118 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f6960 0x7fcdc80f6dc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3196481118 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f56c0 0x7fcdc80f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 -- 172.21.15.84:0/3196481118 >> 172.21.15.84:0/3196481118 conn(0x7fcdc80f0e20 msgr2=0x7fcdc80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 -- 172.21.15.84:0/3196481118 shutdown_connections 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.351+0000 7fcdd069d700 1 -- 172.21.15.84:0/3196481118 wait complete. 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 Processor -- start 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 -- start start 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f6960 0x7fcdc81075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8107b70 con 0x7fcdc80f56c0 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdd069d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc8107cb0 con 0x7fcdc80f6960 2024-01-18T07:34:14.354 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32876/0 (socket says 172.21.15.84:32876) 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 -- 172.21.15.84:0/3699363125 learned_addr learned my addr 172.21.15.84:0/3699363125 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdcdc38700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f6960 0x7fcdc81075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 -- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f6960 msgr2=0x7fcdc81075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f6960 0x7fcdc81075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4005040 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.352+0000 7fcdce439700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fcdbc002700 tx=0x7fcdbc009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdbc0079c0 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc8108650 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdbc00f040 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdbc014860 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc8108c10 con 0x7fcdc80f56c0 2024-01-18T07:34:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.353+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcdc805dd30 con 0x7fcdc80f56c0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcdbc01b060 con 0x7fcdc80f56c0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 0x7fcdb405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcdbc075eb0 con 0x7fcdc80f56c0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdcdc38700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 0x7fcdb405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 0x7fcdb4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdb4064b10 con 0x7fcdb4062030 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.354+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdbc0762b0 con 0x7fcdc80f56c0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.355+0000 7fcdcdc38700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 0x7fcdb405e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcdc4009f50 tx=0x7fcdc4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.355+0000 7fcdcec3a700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 0x7fcdb4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.355+0000 7fcdcec3a700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 0x7fcdb4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:14.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.356+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== osd.3 v2:172.21.15.106:6800/3853148779 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7fcdb4064b10 con 0x7fcdb4062030 2024-01-18T07:34:14.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.370+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 --> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcdc81090d0 con 0x7fcdb4062030 2024-01-18T07:34:14.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.370+0000 7fcdbaffd700 1 -- 172.21.15.84:0/3699363125 <== osd.3 v2:172.21.15.106:6800/3853148779 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcdc81090d0 con 0x7fcdb4062030 2024-01-18T07:34:14.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 msgr2=0x7fcdb4064490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:14.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 0x7fcdb4064490 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 msgr2=0x7fcdb405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 0x7fcdb405e760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcdc4009f50 tx=0x7fcdc4009b30).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 msgr2=0x7fcdc81070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fcdbc002700 tx=0x7fcdbc009d80).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 shutdown_connections 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fcdc80f56c0 0x7fcdc81070d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:6800/3853148779,v1:172.21.15.106:6801/3853148779] conn(0x7fcdb4062030 0x7fcdb4064490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcdb405c2a0 0x7fcdb405e760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 --2- 172.21.15.84:0/3699363125 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdc80f6960 0x7fcdc81075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 >> 172.21.15.84:0/3699363125 conn(0x7fcdc80f0e20 msgr2=0x7fcdc80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 shutdown_connections 2024-01-18T07:34:14.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:14.371+0000 7fcdd069d700 1 -- 172.21.15.84:0/3699363125 wait complete. 2024-01-18T07:34:14.749 INFO:teuthology.orchestra.run.smithi084.stdout:107374182471 2024-01-18T07:34:14.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-01-18T07:34:14.969 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:15.003016+0000 mgr.smithi084.sihbqg (mgr.14162) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:15.003016+0000 mgr.smithi084.sihbqg (mgr.14162) 498 : cluster [DBG] pgmap v466: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:15.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 -- 172.21.15.84:0/1759847820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f4aa0 msgr2=0x7f347c0f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:15.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 --2- 172.21.15.84:0/1759847820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f4aa0 0x7f347c0f4ec0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f346c004660 tx=0x7f346c009b30).stop 2024-01-18T07:34:15.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 -- 172.21.15.84:0/1759847820 shutdown_connections 2024-01-18T07:34:15.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 --2- 172.21.15.84:0/1759847820 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f5440 0x7f347c0f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 --2- 172.21.15.84:0/1759847820 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f4aa0 0x7f347c0f4ec0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 -- 172.21.15.84:0/1759847820 >> 172.21.15.84:0/1759847820 conn(0x7f347c0f05e0 msgr2=0x7f347c0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 -- 172.21.15.84:0/1759847820 shutdown_connections 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.952+0000 7f3481936700 1 -- 172.21.15.84:0/1759847820 wait complete. 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 Processor -- start 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 -- start start 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f4aa0 0x7f347c107160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c107c90 con 0x7f347c0f5440 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f3481936700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347c108540 con 0x7f347c0f4aa0 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f347a7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f347a7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:32890/0 (socket says 172.21.15.84:32890) 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f347a7fc700 1 -- 172.21.15.84:0/1002557745 learned_addr learned my addr 172.21.15.84:0/1002557745 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:15.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.953+0000 7f347affd700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f4aa0 0x7f347c107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f347a7fc700 1 -- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f4aa0 msgr2=0x7f347c107160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f347a7fc700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f4aa0 0x7f347c107160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f347a7fc700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c005040 con 0x7f347c0f5440 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f347a7fc700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f34700093f0 tx=0x7f3470006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3470013640 con 0x7f347c0f5440 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3470013c50 con 0x7f347c0f5440 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c108820 con 0x7f347c0f5440 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3470012c80 con 0x7f347c0f5440 2024-01-18T07:34:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.954+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c108de0 con 0x7f347c0f5440 2024-01-18T07:34:15.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.955+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f347c05dcf0 con 0x7f347c0f5440 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.955+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f347001a060 con 0x7f347c0f5440 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.955+0000 7f34637fe700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 0x7f346405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.955+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3470075b80 con 0x7f347c0f5440 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.955+0000 7f347affd700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 0x7f346405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f34637fe700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 0x7f34640644e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3464064b60 con 0x7f3464062080 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34700781f0 con 0x7f347c0f5440 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f347affd700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 0x7f346405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f346c004630 tx=0x7f346c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f347b7fe700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 0x7f34640644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.956+0000 7f347b7fe700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 0x7f34640644e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:15.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.957+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== osd.4 v2:172.21.15.106:6808/4131077435 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f3464064b60 con 0x7f3464062080 2024-01-18T07:34:15.971 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.971+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 --> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f347c109320 con 0x7f3464062080 2024-01-18T07:34:15.972 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.971+0000 7f34637fe700 1 -- 172.21.15.84:0/1002557745 <== osd.4 v2:172.21.15.106:6808/4131077435 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f347c109320 con 0x7f3464062080 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 msgr2=0x7f34640644e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 0x7f34640644e0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 msgr2=0x7f346405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 0x7f346405e760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f346c004630 tx=0x7f346c009b30).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 msgr2=0x7f347c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f34700093f0 tx=0x7f3470006ec0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 shutdown_connections 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:6808/4131077435,v1:172.21.15.106:6809/4131077435] conn(0x7f3464062080 0x7f34640644e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f346405c2a0 0x7f346405e760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f347c0f4aa0 0x7f347c107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 --2- 172.21.15.84:0/1002557745 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f347c0f5440 0x7f347c107650 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:15.973 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 >> 172.21.15.84:0/1002557745 conn(0x7f347c0f05e0 msgr2=0x7f347c0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:15.974 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 shutdown_connections 2024-01-18T07:34:15.974 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:15.972+0000 7f3481936700 1 -- 172.21.15.84:0/1002557745 wait complete. 2024-01-18T07:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:133143986209 2024-01-18T07:34:16.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-01-18T07:34:16.607 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:17.003371+0000 mgr.smithi084.sihbqg (mgr.14162) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:17.003371+0000 mgr.smithi084.sihbqg (mgr.14162) 499 : cluster [DBG] pgmap v467: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 -- 172.21.15.84:0/3298639766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 msgr2=0x7f15ec0f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 --2- 172.21.15.84:0/3298639766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec0f5930 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f15dc004660 tx=0x7f15dc009b30).stop 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 -- 172.21.15.84:0/3298639766 shutdown_connections 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 --2- 172.21.15.84:0/3298639766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec0f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 --2- 172.21.15.84:0/3298639766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec0f5930 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 -- 172.21.15.84:0/3298639766 >> 172.21.15.84:0/3298639766 conn(0x7f15ec0f0c70 msgr2=0x7f15ec0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 -- 172.21.15.84:0/3298639766 shutdown_connections 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.593+0000 7f15f0f51700 1 -- 172.21.15.84:0/3298639766 wait complete. 2024-01-18T07:34:17.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 Processor -- start 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 -- start start 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec106390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec1069d0 con 0x7f15ec0f5510 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15f0f51700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec107140 con 0x7f15ec0f67b0 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15eb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15eb7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54760/0 (socket says 172.21.15.84:54760) 2024-01-18T07:34:17.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15eb7fe700 1 -- 172.21.15.84:0/769305455 learned_addr learned my addr 172.21.15.84:0/769305455 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.594+0000 7f15eaffd700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec106390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15eb7fe700 1 -- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 msgr2=0x7f15ec106390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15eb7fe700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec106390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15eb7fe700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc005040 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15eaffd700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15eb7fe700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f15dc009f50 tx=0x7f15dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15dc018070 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15dc013410 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec1073c0 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15dc01c620 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.595+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec107920 con 0x7f15ec0f5510 2024-01-18T07:34:17.597 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.596+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15ec05db80 con 0x7f15ec0f5510 2024-01-18T07:34:17.598 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.596+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f15dc01d420 con 0x7f15ec0f5510 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.596+0000 7f15d3fff700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 0x7f15d405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f15dc0781f0 con 0x7f15ec0f5510 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15eaffd700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 0x7f15d405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15d3fff700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 0x7f15d40644a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15d4064b20 con 0x7f15d4062040 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15dc0785f0 con 0x7f15ec0f5510 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15eaffd700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 0x7f15d405e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f15e000b830 tx=0x7f15e0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15ebfff700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 0x7f15d40644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.597+0000 7f15ebfff700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 0x7f15d40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:17.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.598+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== osd.5 v2:172.21.15.106:6816/408419143 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23065 (crc 0 0 0) 0x7f15d4064b20 con 0x7f15d4062040 2024-01-18T07:34:17.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.612+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 --> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f15ec107e40 con 0x7f15d4062040 2024-01-18T07:34:17.613 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15d3fff700 1 -- 172.21.15.84:0/769305455 <== osd.5 v2:172.21.15.106:6816/408419143 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f15ec107e40 con 0x7f15d4062040 2024-01-18T07:34:17.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 msgr2=0x7f15d40644a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:17.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 0x7f15d40644a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 msgr2=0x7f15d405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:17.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 0x7f15d405e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f15e000b830 tx=0x7f15e0008040).stop 2024-01-18T07:34:17.614 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 msgr2=0x7f15ec105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f15dc009f50 tx=0x7f15dc009b30).stop 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 shutdown_connections 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:6816/408419143,v1:172.21.15.106:6817/408419143] conn(0x7f15d4062040 0x7f15d40644a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15ec0f5510 0x7f15ec105ea0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f15d405c2b0 0x7f15d405e770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 --2- 172.21.15.84:0/769305455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15ec0f67b0 0x7f15ec106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 >> 172.21.15.84:0/769305455 conn(0x7f15ec0f0c70 msgr2=0x7f15ec0fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 shutdown_connections 2024-01-18T07:34:17.615 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:17.613+0000 7f15f0f51700 1 -- 172.21.15.84:0/769305455 wait complete. 2024-01-18T07:34:18.049 INFO:teuthology.orchestra.run.smithi084.stdout:154618822664 2024-01-18T07:34:18.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-01-18T07:34:18.270 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:19.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 -- 172.21.15.84:0/433237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 msgr2=0x7fd27406a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:19.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 --2- 172.21.15.84:0/433237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd27406a7b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd264004660 tx=0x7fd264009b30).stop 2024-01-18T07:34:19.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 -- 172.21.15.84:0/433237336 shutdown_connections 2024-01-18T07:34:19.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 --2- 172.21.15.84:0/433237336 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd27406a7b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:19.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 --2- 172.21.15.84:0/433237336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd27406b940 0x7fd274069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:19.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 -- 172.21.15.84:0/433237336 >> 172.21.15.84:0/433237336 conn(0x7fd2740f1f20 msgr2=0x7fd2740f4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:19.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 -- 172.21.15.84:0/433237336 shutdown_connections 2024-01-18T07:34:19.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 -- 172.21.15.84:0/433237336 wait complete. 2024-01-18T07:34:19.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.251+0000 7fd27943c700 1 Processor -- start 2024-01-18T07:34:19.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 -- start start 2024-01-18T07:34:19.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd27406b940 0x7fd274109770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd274109db0 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27410a660 con 0x7fd27406b940 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54774/0 (socket says 172.21.15.84:54774) 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 -- 172.21.15.84:0/2216908972 learned_addr learned my addr 172.21.15.84:0/2216908972 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 -- 172.21.15.84:0/2216908972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd27406b940 msgr2=0x7fd274109770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd27406b940 0x7fd274109770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 -- 172.21.15.84:0/2216908972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264005040 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd272ffd700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fd26800b410 tx=0x7fd268006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2680125f0 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27410a940 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd268012c00 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd268002aa0 con 0x7fd27406a330 2024-01-18T07:34:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.252+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27410af00 con 0x7fd27406a330 2024-01-18T07:34:19.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.253+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd274059100 con 0x7fd27406a330 2024-01-18T07:34:19.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.254+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd26800f030 con 0x7fd27406a330 2024-01-18T07:34:19.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.254+0000 7fd25b7fe700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 0x7fd25c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:19.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.254+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd268076060 con 0x7fd27406a330 2024-01-18T07:34:19.256 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.254+0000 7fd2727fc700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 0x7fd25c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:19.256 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.254+0000 7fd2727fc700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 0x7fd25c05e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd264009f50 tx=0x7fd264009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:19.258 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.258+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd268049000 con 0x7fd27406a330 2024-01-18T07:34:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:19.003745+0000 mgr.smithi084.sihbqg (mgr.14162) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:19.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.408+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd274101310 con 0x7fd27406a330 2024-01-18T07:34:19.409 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.409+0000 7fd25b7fe700 1 -- 172.21.15.84:0/2216908972 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd26804c620 con 0x7fd27406a330 2024-01-18T07:34:19.409 INFO:teuthology.orchestra.run.smithi084.stdout:38654705786 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.410+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 msgr2=0x7fd25c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.410+0000 7fd27943c700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 0x7fd25c05e760 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fd264009f50 tx=0x7fd264009b30).stop 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.410+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 msgr2=0x7fd274109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.410+0000 7fd27943c700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fd26800b410 tx=0x7fd268006ef0).stop 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 shutdown_connections 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd27406a330 0x7fd274109280 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd25c05c2a0 0x7fd25c05e760 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 --2- 172.21.15.84:0/2216908972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd27406b940 0x7fd274109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:19.411 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 >> 172.21.15.84:0/2216908972 conn(0x7fd2740f1f20 msgr2=0x7fd2740fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:19.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 shutdown_connections 2024-01-18T07:34:19.412 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:19.411+0000 7fd27943c700 1 -- 172.21.15.84:0/2216908972 wait complete. 2024-01-18T07:34:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:19.003745+0000 mgr.smithi084.sihbqg (mgr.14162) 500 : cluster [DBG] pgmap v468: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:19.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705784 got 38654705786 for osd.0 2024-01-18T07:34:19.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-01-18T07:34:20.057 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:20.351 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:19.409948+0000 mon.smithi084 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.84:0/2216908972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-18T07:34:20.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:19.409948+0000 mon.smithi084 (mon.0) 496 : audit [DBG] from='client.? 172.21.15.84:0/2216908972' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-01-18T07:34:21.052 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 -- 172.21.15.84:0/887455824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 msgr2=0x7f248c0fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 --2- 172.21.15.84:0/887455824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0fbd70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f247c004660 tx=0x7f247c009b30).stop 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 -- 172.21.15.84:0/887455824 shutdown_connections 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 --2- 172.21.15.84:0/887455824 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0fbd70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 --2- 172.21.15.84:0/887455824 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 0x7f248c0f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 -- 172.21.15.84:0/887455824 >> 172.21.15.84:0/887455824 conn(0x7f248c0f0e20 msgr2=0x7f248c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 -- 172.21.15.84:0/887455824 shutdown_connections 2024-01-18T07:34:21.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.052+0000 7f2491834700 1 -- 172.21.15.84:0/887455824 wait complete. 2024-01-18T07:34:21.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 Processor -- start 2024-01-18T07:34:21.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 -- start start 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 0x7f248c0f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0f5800 con 0x7f248c0f9920 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f2491834700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0f5940 con 0x7f248c060870 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f248a7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f248a7fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54802/0 (socket says 172.21.15.84:54802) 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f248a7fc700 1 -- 172.21.15.84:0/2301615812 learned_addr learned my addr 172.21.15.84:0/2301615812 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f248affd700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 0x7f248c0f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.053+0000 7f248a7fc700 1 -- 172.21.15.84:0/2301615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 msgr2=0x7f248c0f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f248a7fc700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 0x7f248c0f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f248a7fc700 1 -- 172.21.15.84:0/2301615812 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c005040 con 0x7f248c0f9920 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f248a7fc700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f247c009830 tx=0x7f247c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f247c018070 con 0x7f248c0f9920 2024-01-18T07:34:21.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f247c007ba0 con 0x7f248c0f9920 2024-01-18T07:34:21.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c0f5bc0 con 0x7f248c0f9920 2024-01-18T07:34:21.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f247c01de20 con 0x7f248c0f9920 2024-01-18T07:34:21.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.054+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c0f6120 con 0x7f248c0f9920 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.055+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c045ad0 con 0x7f248c0f9920 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.055+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f247c027440 con 0x7f248c0f9920 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.055+0000 7f24737fe700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 0x7f247405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.055+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f247c077720 con 0x7f248c0f9920 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.055+0000 7f248affd700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 0x7f247405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:21.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.056+0000 7f248affd700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 0x7f247405e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f248c0f6bb0 tx=0x7f248000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:21.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.059+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f247c04a720 con 0x7f248c0f9920 2024-01-18T07:34:21.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.209+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f248c027a00 con 0x7f248c0f9920 2024-01-18T07:34:21.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.209+0000 7f24737fe700 1 -- 172.21.15.84:0/2301615812 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f247c04dd40 con 0x7f248c0f9920 2024-01-18T07:34:21.210 INFO:teuthology.orchestra.run.smithi084.stdout:55834574969 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 msgr2=0x7f247405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 0x7f247405e760 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f248c0f6bb0 tx=0x7f248000b040).stop 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 msgr2=0x7f248c0f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f247c009830 tx=0x7f247c009b30).stop 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 shutdown_connections 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f247405c2a0 0x7f247405e760 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f248c060870 0x7f248c0f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 --2- 172.21.15.84:0/2301615812 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f248c0f9920 0x7f248c0f52a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 >> 172.21.15.84:0/2301615812 conn(0x7f248c0f0e20 msgr2=0x7f248c0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 shutdown_connections 2024-01-18T07:34:21.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:21.211+0000 7f2491834700 1 -- 172.21.15.84:0/2301615812 wait complete. 2024-01-18T07:34:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:21.004093+0000 mgr.smithi084.sihbqg (mgr.14162) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:21.004093+0000 mgr.smithi084.sihbqg (mgr.14162) 501 : cluster [DBG] pgmap v469: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:21.625 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574967 got 55834574969 for osd.1 2024-01-18T07:34:21.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-01-18T07:34:21.845 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:22.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:21.210633+0000 mon.smithi084 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.84:0/2301615812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-18T07:34:22.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:21.210633+0000 mon.smithi084 (mon.0) 497 : audit [DBG] from='client.? 172.21.15.84:0/2301615812' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 -- 172.21.15.84:0/3828628457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f67b0 msgr2=0x7f39980f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 --2- 172.21.15.84:0/3828628457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f67b0 0x7f39980f6c10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f3994004660 tx=0x7f3994009b30).stop 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 -- 172.21.15.84:0/3828628457 shutdown_connections 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 --2- 172.21.15.84:0/3828628457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f67b0 0x7f39980f6c10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 --2- 172.21.15.84:0/3828628457 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39980f5510 0x7f39980f5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:22.916 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 -- 172.21.15.84:0/3828628457 >> 172.21.15.84:0/3828628457 conn(0x7f39980f0c70 msgr2=0x7f39980f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 -- 172.21.15.84:0/3828628457 shutdown_connections 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.914+0000 7f399e02d700 1 -- 172.21.15.84:0/3828628457 wait complete. 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 Processor -- start 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 -- start start 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39980f67b0 0x7f399810c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39980609b0 con 0x7f39980f5510 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399e02d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3998060af0 con 0x7f39980f67b0 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54822/0 (socket says 172.21.15.84:54822) 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 -- 172.21.15.84:0/1657246087 learned_addr learned my addr 172.21.15.84:0/1657246087 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 -- 172.21.15.84:0/1657246087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39980f67b0 msgr2=0x7f399810c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39980f67b0 0x7f399810c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 -- 172.21.15.84:0/1657246087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994005040 con 0x7f39980f5510 2024-01-18T07:34:22.917 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.915+0000 7f399d02b700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f39880093f0 tx=0x7f3988006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:22.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.916+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3988013640 con 0x7f39980f5510 2024-01-18T07:34:22.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.916+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3988009e40 con 0x7f39980f5510 2024-01-18T07:34:22.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.916+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399810c750 con 0x7f39980f5510 2024-01-18T07:34:22.918 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.916+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f398800dbf0 con 0x7f39980f5510 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.916+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399810cd10 con 0x7f39980f5510 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.917+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3998045ad0 con 0x7f39980f5510 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.917+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f398801a060 con 0x7f39980f5510 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.917+0000 7f398dffb700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 0x7f398405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.917+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3988075ad0 con 0x7f39980f5510 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.917+0000 7f399c82a700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 0x7f398405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:22.919 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.918+0000 7f399c82a700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 0x7f398405e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3994009f50 tx=0x7f3994009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:22.921 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:22.921+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3988048af0 con 0x7f39980f5510 2024-01-18T07:34:23.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.071+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3998027a00 con 0x7f39980f5510 2024-01-18T07:34:23.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.071+0000 7f398dffb700 1 -- 172.21.15.84:0/1657246087 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f398804c110 con 0x7f39980f5510 2024-01-18T07:34:23.072 INFO:teuthology.orchestra.run.smithi084.stdout:73014444150 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 msgr2=0x7f398405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 0x7f398405e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3994009f50 tx=0x7f3994009b30).stop 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 msgr2=0x7f3998109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f39880093f0 tx=0x7f3988006ec0).stop 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 shutdown_connections 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f39980f5510 0x7f3998109c40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:23.073 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f398405c2b0 0x7f398405e770 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:23.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 --2- 172.21.15.84:0/1657246087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f39980f67b0 0x7f399810c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:23.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 >> 172.21.15.84:0/1657246087 conn(0x7f39980f0c70 msgr2=0x7f39980fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:23.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 shutdown_connections 2024-01-18T07:34:23.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:23.073+0000 7f399e02d700 1 -- 172.21.15.84:0/1657246087 wait complete. 2024-01-18T07:34:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:23.004442+0000 mgr.smithi084.sihbqg (mgr.14162) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:23.072195+0000 mon.smithi084 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.84:0/1657246087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-18T07:34:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:23.004442+0000 mgr.smithi084.sihbqg (mgr.14162) 502 : cluster [DBG] pgmap v470: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:23.072195+0000 mon.smithi084 (mon.0) 498 : audit [DBG] from='client.? 172.21.15.84:0/1657246087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-01-18T07:34:23.497 INFO:tasks.cephadm.ceph_manager.ceph:need seq 73014444148 got 73014444150 for osd.2 2024-01-18T07:34:23.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-01-18T07:34:23.717 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:24.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.767+0000 7f54566e2700 1 -- 172.21.15.84:0/1000867074 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 msgr2=0x7f54500f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:24.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.767+0000 7f54566e2700 1 --2- 172.21.15.84:0/1000867074 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54500f6be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f5440004660 tx=0x7f5440009b30).stop 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- 172.21.15.84:0/1000867074 shutdown_connections 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 --2- 172.21.15.84:0/1000867074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 0x7f54500f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 --2- 172.21.15.84:0/1000867074 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54500f6be0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- 172.21.15.84:0/1000867074 >> 172.21.15.84:0/1000867074 conn(0x7f54500f1f20 msgr2=0x7f54500f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- 172.21.15.84:0/1000867074 shutdown_connections 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- 172.21.15.84:0/1000867074 wait complete. 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 Processor -- start 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- start start 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 0x7f54501075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5450107c00 con 0x7f54500f67c0 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.768+0000 7f54566e2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54501084b0 con 0x7f54500f7a60 2024-01-18T07:34:24.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54848/0 (socket says 172.21.15.84:54848) 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 -- 172.21.15.84:0/2413175192 learned_addr learned my addr 172.21.15.84:0/2413175192 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544f7fe700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 0x7f54501075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 -- 172.21.15.84:0/2413175192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 msgr2=0x7f54501075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 0x7f54501075c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 -- 172.21.15.84:0/2413175192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5440005040 con 0x7f54500f67c0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544ffff700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f5440009f50 tx=0x7f5440009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5440018070 con 0x7f54500f67c0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5450108730 con 0x7f54500f67c0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5440013410 con 0x7f54500f67c0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f544001c620 con 0x7f54500f67c0 2024-01-18T07:34:24.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.769+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5450108c90 con 0x7f54500f67c0 2024-01-18T07:34:24.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.770+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5450059100 con 0x7f54500f67c0 2024-01-18T07:34:24.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.770+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f544001d420 con 0x7f54500f67c0 2024-01-18T07:34:24.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.771+0000 7f544cff9700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 0x7f543c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:24.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.771+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f54400779f0 con 0x7f54500f67c0 2024-01-18T07:34:24.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.771+0000 7f544f7fe700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 0x7f543c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:24.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.771+0000 7f544f7fe700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 0x7f543c05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f543800a6f0 tx=0x7f5438008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:24.775 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.775+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5440079e60 con 0x7f54500f67c0 2024-01-18T07:34:24.927 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.927+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5450045ad0 con 0x7f54500f67c0 2024-01-18T07:34:24.927 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.927+0000 7f544cff9700 1 -- 172.21.15.84:0/2413175192 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f544004a9f0 con 0x7f54500f67c0 2024-01-18T07:34:24.927 INFO:teuthology.orchestra.run.smithi084.stdout:107374182473 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 msgr2=0x7f543c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 0x7f543c05e760 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f543800a6f0 tx=0x7f5438008040).stop 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 msgr2=0x7f54501070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f5440009f50 tx=0x7f5440009b30).stop 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 shutdown_connections 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f54500f67c0 0x7f54501070d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f543c05c2a0 0x7f543c05e760 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 --2- 172.21.15.84:0/2413175192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f54500f7a60 0x7f54501075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.928+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 >> 172.21.15.84:0/2413175192 conn(0x7f54500f1f20 msgr2=0x7f54500fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:24.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.929+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 shutdown_connections 2024-01-18T07:34:24.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:24.929+0000 7f54566e2700 1 -- 172.21.15.84:0/2413175192 wait complete. 2024-01-18T07:34:25.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:24.928037+0000 mon.smithi084 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.84:0/2413175192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-18T07:34:25.304 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:24.928037+0000 mon.smithi084 (mon.0) 499 : audit [DBG] from='client.? 172.21.15.84:0/2413175192' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-01-18T07:34:25.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182471 got 107374182473 for osd.3 2024-01-18T07:34:25.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-01-18T07:34:25.575 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:26.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:25.004773+0000 mgr.smithi084.sihbqg (mgr.14162) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:26.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:25.004773+0000 mgr.smithi084.sihbqg (mgr.14162) 503 : cluster [DBG] pgmap v471: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- 172.21.15.84:0/2420941467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240607d0 msgr2=0x7f48240f9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 --2- 172.21.15.84:0/2420941467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240607d0 0x7f48240f9440 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f4818004660 tx=0x7f4818009b30).stop 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- 172.21.15.84:0/2420941467 shutdown_connections 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 --2- 172.21.15.84:0/2420941467 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240f9930 0x7f48240fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 --2- 172.21.15.84:0/2420941467 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240607d0 0x7f48240f9440 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- 172.21.15.84:0/2420941467 >> 172.21.15.84:0/2420941467 conn(0x7f48240f0e20 msgr2=0x7f48240f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- 172.21.15.84:0/2420941467 shutdown_connections 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- 172.21.15.84:0/2420941467 wait complete. 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 Processor -- start 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.584+0000 7f4828acd700 1 -- start start 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 0x7f48241070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:26.586 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4824107b60 con 0x7f48240f9930 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4824107ca0 con 0x7f48240607d0 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33324/0 (socket says 172.21.15.84:33324) 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 -- 172.21.15.84:0/3980944143 learned_addr learned my addr 172.21.15.84:0/3980944143 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f482259c700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 0x7f48241070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 -- 172.21.15.84:0/3980944143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 msgr2=0x7f48241070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 0x7f48241070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 -- 172.21.15.84:0/3980944143 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818005040 con 0x7f48240f9930 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f482259c700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 0x7f48241070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4821d9b700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f481400c510 tx=0x7f4814007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f481400f650 con 0x7f48240f9930 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f481400fc60 con 0x7f48240f9930 2024-01-18T07:34:26.587 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4824108640 con 0x7f48240f9930 2024-01-18T07:34:26.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4814012c80 con 0x7f48240f9930 2024-01-18T07:34:26.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.585+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4824108c00 con 0x7f48240f9930 2024-01-18T07:34:26.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.586+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4824045ad0 con 0x7f48240f9930 2024-01-18T07:34:26.588 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.587+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4814010070 con 0x7f48240f9930 2024-01-18T07:34:26.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.587+0000 7f4812ffd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 0x7f480c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:26.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.587+0000 7f482259c700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 0x7f480c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:26.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.587+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f48140758d0 con 0x7f48240f9930 2024-01-18T07:34:26.589 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.587+0000 7f482259c700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 0x7f480c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4818004660 tx=0x7f4818009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:26.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.591+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4814048870 con 0x7f48240f9930 2024-01-18T07:34:26.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.738+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4824027a00 con 0x7f48240f9930 2024-01-18T07:34:26.738 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.738+0000 7f4812ffd700 1 -- 172.21.15.84:0/3980944143 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f481404be90 con 0x7f48240f9930 2024-01-18T07:34:26.738 INFO:teuthology.orchestra.run.smithi084.stdout:133143986211 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.739+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 msgr2=0x7f480c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.739+0000 7f4828acd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 0x7f480c05e760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4818004660 tx=0x7f4818009b30).stop 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 msgr2=0x7f48241075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f481400c510 tx=0x7f4814007df0).stop 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 shutdown_connections 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f480c05c2a0 0x7f480c05e760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f48240607d0 0x7f48241070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.740 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 --2- 172.21.15.84:0/3980944143 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f48240f9930 0x7f48241075b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:26.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 >> 172.21.15.84:0/3980944143 conn(0x7f48240f0e20 msgr2=0x7f48240f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:26.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 shutdown_connections 2024-01-18T07:34:26.741 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:26.740+0000 7f4828acd700 1 -- 172.21.15.84:0/3980944143 wait complete. 2024-01-18T07:34:27.129 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986209 got 133143986211 for osd.4 2024-01-18T07:34:27.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-01-18T07:34:27.351 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:27.381 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:26.739218+0000 mon.smithi084 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.84:0/3980944143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-18T07:34:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:27.005127+0000 mgr.smithi084.sihbqg (mgr.14162) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:26.739218+0000 mon.smithi084 (mon.0) 500 : audit [DBG] from='client.? 172.21.15.84:0/3980944143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-01-18T07:34:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:27.005127+0000 mgr.smithi084.sihbqg (mgr.14162) 504 : cluster [DBG] pgmap v472: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:28.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 -- 172.21.15.84:0/3684021801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd984060810 msgr2=0x7fd9840f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:28.288 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/3684021801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd984060810 0x7fd9840f9400 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fd974004660 tx=0x7fd974009b30).stop 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 -- 172.21.15.84:0/3684021801 shutdown_connections 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/3684021801 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9840f9980 0x7fd9840fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/3684021801 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd984060810 0x7fd9840f9400 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 -- 172.21.15.84:0/3684021801 >> 172.21.15.84:0/3684021801 conn(0x7fd9840f0e20 msgr2=0x7fd9840f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 -- 172.21.15.84:0/3684021801 shutdown_connections 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 -- 172.21.15.84:0/3684021801 wait complete. 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.286+0000 7fd98aa0f700 1 Processor -- start 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 -- start start 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984060810 0x7fd9841070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984107c10 con 0x7fd9840f9980 2024-01-18T07:34:28.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9841084c0 con 0x7fd984060810 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33340/0 (socket says 172.21.15.84:33340) 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 -- 172.21.15.84:0/4071006871 learned_addr learned my addr 172.21.15.84:0/4071006871 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd983fff700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984060810 0x7fd9841070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 -- 172.21.15.84:0/4071006871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984060810 msgr2=0x7fd9841070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984060810 0x7fd9841070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 -- 172.21.15.84:0/4071006871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974005040 con 0x7fd9840f9980 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd9837fe700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd9780093f0 tx=0x7fd978006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd978009ea0 con 0x7fd9840f9980 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9841087a0 con 0x7fd9840f9980 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.287+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd978013af0 con 0x7fd9840f9980 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.288+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd97800dad0 con 0x7fd9840f9980 2024-01-18T07:34:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.288+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd984108d60 con 0x7fd9840f9980 2024-01-18T07:34:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.288+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd984059100 con 0x7fd9840f9980 2024-01-18T07:34:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.289+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd97801a060 con 0x7fd9840f9980 2024-01-18T07:34:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.289+0000 7fd980ff9700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 0x7fd96c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.289+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd978075bb0 con 0x7fd9840f9980 2024-01-18T07:34:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.289+0000 7fd983fff700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 0x7fd96c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:28.292 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.290+0000 7fd983fff700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 0x7fd96c05e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd974009d60 tx=0x7fd974009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:28.293 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.293+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd97807c080 con 0x7fd9840f9980 2024-01-18T07:34:28.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.442+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd9840feff0 con 0x7fd9840f9980 2024-01-18T07:34:28.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.442+0000 7fd980ff9700 1 -- 172.21.15.84:0/4071006871 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd978048bb0 con 0x7fd9840f9980 2024-01-18T07:34:28.442 INFO:teuthology.orchestra.run.smithi084.stdout:154618822666 2024-01-18T07:34:28.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 msgr2=0x7fd96c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:28.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 0x7fd96c05e760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd974009d60 tx=0x7fd974009b30).stop 2024-01-18T07:34:28.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 msgr2=0x7fd9841075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:28.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd9780093f0 tx=0x7fd978006ec0).stop 2024-01-18T07:34:28.444 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 shutdown_connections 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd96c05c2a0 0x7fd96c05e760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd984060810 0x7fd9841070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 --2- 172.21.15.84:0/4071006871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd9840f9980 0x7fd9841075d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 >> 172.21.15.84:0/4071006871 conn(0x7fd9840f0e20 msgr2=0x7fd9840f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 shutdown_connections 2024-01-18T07:34:28.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:28.444+0000 7fd98aa0f700 1 -- 172.21.15.84:0/4071006871 wait complete. 2024-01-18T07:34:28.783 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:28.443236+0000 mon.smithi084 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.84:0/4071006871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-18T07:34:28.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822664 got 154618822666 for osd.5 2024-01-18T07:34:28.837 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-01-18T07:34:28.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-18T07:34:28.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:28.443236+0000 mon.smithi084 (mon.0) 501 : audit [DBG] from='client.? 172.21.15.84:0/4071006871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-01-18T07:34:29.055 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:29.875 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:29.005464+0000 mgr.smithi084.sihbqg (mgr.14162) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:29.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:29.005464+0000 mgr.smithi084.sihbqg (mgr.14162) 505 : cluster [DBG] pgmap v473: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:30.066 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.064+0000 7f7d63ac9700 1 -- 172.21.15.84:0/363075770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f6960 msgr2=0x7f7d5c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.064+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/363075770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c0f6dc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f7d4c004660 tx=0x7f7d4c009b30).stop 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- 172.21.15.84:0/363075770 shutdown_connections 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/363075770 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c0f6dc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/363075770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- 172.21.15.84:0/363075770 >> 172.21.15.84:0/363075770 conn(0x7f7d5c0f0e20 msgr2=0x7f7d5c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- 172.21.15.84:0/363075770 shutdown_connections 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- 172.21.15.84:0/363075770 wait complete. 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 Processor -- start 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- start start 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c10c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:30.067 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c060920 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d63ac9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c060a60 con 0x7f7d5c0f6960 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d61865700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d61865700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33372/0 (socket says 172.21.15.84:33372) 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d61865700 1 -- 172.21.15.84:0/1371013373 learned_addr learned my addr 172.21.15.84:0/1371013373 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.065+0000 7f7d61064700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c10c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d61865700 1 -- 172.21.15.84:0/1371013373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f6960 msgr2=0x7f7d5c10c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d61865700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c10c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d61865700 1 -- 172.21.15.84:0/1371013373 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c005040 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d61865700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7d58002700 tx=0x7f7d58009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d580079c0 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c10c8b0 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d5800f040 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d58014860 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.068 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.066+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c10ce40 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.067+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5c059100 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.067+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7d5801b060 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.067+0000 7f7d527fc700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 0x7f7d4805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:30.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.068+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7d58075a40 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.068+0000 7f7d61064700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 0x7f7d4805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:30.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.068+0000 7f7d61064700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 0x7f7d4805e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f7d4c009f50 tx=0x7f7d4c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:30.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.071+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d58048a40 con 0x7f7d5c0f56c0 2024-01-18T07:34:30.217 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.217+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d5c057f10 con 0x7f7d4805c2a0 2024-01-18T07:34:30.219 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.218+0000 7f7d527fc700 1 -- 172.21.15.84:0/1371013373 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27600 (secure 0 0 0) 0x7f7d5c057f10 con 0x7f7d4805c2a0 2024-01-18T07:34:30.219 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.220+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 msgr2=0x7f7d4805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.220+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 0x7f7d4805e760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f7d4c009f50 tx=0x7f7d4c009b30).stop 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.220+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 msgr2=0x7f7d5c109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.220+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7d58002700 tx=0x7f7d58009d80).stop 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 shutdown_connections 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f7d5c0f56c0 0x7f7d5c109e00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f7d4805c2a0 0x7f7d4805e760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 --2- 172.21.15.84:0/1371013373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d5c0f6960 0x7f7d5c10c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:30.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 >> 172.21.15.84:0/1371013373 conn(0x7f7d5c0f0e20 msgr2=0x7f7d5c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:30.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 shutdown_connections 2024-01-18T07:34:30.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:30.221+0000 7f7d63ac9700 1 -- 172.21.15.84:0/1371013373 wait complete. 2024-01-18T07:34:30.222 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-01-18T07:34:30.654 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-01-18T07:34:29.005317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31112,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439800,"statfs":{"total":575970213888,"available":575938355200,"internally_reserved":0,"allocated":1990656,"data_stored":760032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2024-01-18T07:33:52.797273+0000","last_change":"2024-01-18T07:31:42.974429+0000","last_active":"2024-01-18T07:33:52.797273+0000","last_peered":"2024-01-18T07:33:52.797273+0000","last_clean":"2024-01-18T07:33:52.797273+0000","last_became_active":"2024-01-18T07:31:42.974060+0000","last_became_peered":"2024-01-18T07:31:42.974060+0000","last_unstale":"2024-01-18T07:33:52.797273+0000","last_undegraded":"2024-01-18T07:33:52.797273+0000","last_fullsized":"2024-01-18T07:33:52.797273+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T07:28:26.574375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T07:28:26.574375+0000","last_clean_scrub_stamp":"2024-01-18T07:28:26.574375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.621,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.66300000000000003,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.38400000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.61299999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.503}]},{"osd":2,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.68100000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.624,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.376,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.437,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.42699999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.68999999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71599999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.505}]},{"osd":1,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69199999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53700000000000003,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.315,"15min":0.315},"max":{"1min":0.629,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.496}]},{"osd":2,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.56799999999999995},"min":{"1min":0.39700000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.64400000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.41499999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.73699999999999999,"5min":0.755,"15min":0.755},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 18 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41499999999999998,"15min":0.42399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.499,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52500000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":2,"up_from":17,"seq":73014444152,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.34100000000000003},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.504,"5min":0.504,"15min":0.51000000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.23799999999999999,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.497},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 18 07:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.36299999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.46500000000000002,"5min":0.504,"15min":0.51300000000000001},"last":0.223},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.54700000000000004},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Jan 18 07:33:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.63400000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66800000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.439}]},{"osd":4,"last update":"Thu Jan 18 07:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.67500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71499999999999997,"5min":0.747,"15min":0.747},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705788,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46100000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35199999999999998,"15min":0.34699999999999998},"min":{"1min":0.26400000000000001,"5min":0.252,"15min":0.23000000000000001},"max":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.43099999999999999}]},{"osd":2,"last update":"Thu Jan 18 07:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.377},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.498,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.42099999999999999}]},{"osd":3,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48499999999999999,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.504},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65300000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35399999999999998,"15min":0.33100000000000002},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.35699999999999998,"15min":0.33800000000000002},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.34699999999999998}]},{"osd":2,"last update":"Thu Jan 18 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.38600000000000001,"15min":0.36199999999999999},"min":{"1min":0.254,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.35999999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313}]},{"osd":3,"last update":"Thu Jan 18 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.66900000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 18 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-18T07:34:30.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph pg dump --format=json 2024-01-18T07:34:30.874 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:30.218439+0000 mgr.smithi084.sihbqg (mgr.14162) 506 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T07:34:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:31.005979+0000 mgr.smithi084.sihbqg (mgr.14162) 507 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:30.218439+0000 mgr.smithi084.sihbqg (mgr.14162) 506 : audit [DBG] from='client.14518 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T07:34:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:31.005979+0000 mgr.smithi084.sihbqg (mgr.14162) 507 : cluster [DBG] pgmap v474: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:31.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 -- 172.21.15.84:0/2780736248 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 msgr2=0x7f14780f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:31.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 --2- 172.21.15.84:0/2780736248 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f14780f7ed0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f1464004660 tx=0x7f1464009b30).stop 2024-01-18T07:34:31.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 -- 172.21.15.84:0/2780736248 shutdown_connections 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 --2- 172.21.15.84:0/2780736248 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f14780f7ed0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 --2- 172.21.15.84:0/2780736248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 0x7f14780f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 -- 172.21.15.84:0/2780736248 >> 172.21.15.84:0/2780736248 conn(0x7f14780f1f50 msgr2=0x7f14780f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.889+0000 7f147ebf0700 1 -- 172.21.15.84:0/2780736248 shutdown_connections 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 -- 172.21.15.84:0/2780736248 wait complete. 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 Processor -- start 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 -- start start 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 0x7f1478107010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478107ab0 con 0x7f14780f7a70 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f147ebf0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478107bf0 con 0x7f14780f67d0 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f1477fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:31.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f1477fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33394/0 (socket says 172.21.15.84:33394) 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.890+0000 7f1477fff700 1 -- 172.21.15.84:0/735796305 learned_addr learned my addr 172.21.15.84:0/735796305 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f147c98c700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 0x7f1478107010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f1477fff700 1 -- 172.21.15.84:0/735796305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 msgr2=0x7f1478107010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f1477fff700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 0x7f1478107010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f1477fff700 1 -- 172.21.15.84:0/735796305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464005040 con 0x7f14780f7a70 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f1477fff700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1464000c00 tx=0x7f146400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1464019070 con 0x7f14780f7a70 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1464007e70 con 0x7f14780f7a70 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1478108530 con 0x7f14780f7a70 2024-01-18T07:34:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1464014c90 con 0x7f14780f7a70 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.891+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1478108a90 con 0x7f14780f7a70 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.892+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1478045ad0 con 0x7f14780f7a70 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.892+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1464025070 con 0x7f14780f7a70 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.893+0000 7f14757fa700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 0x7f146805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.893+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1464076be0 con 0x7f14780f7a70 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.893+0000 7f147c98c700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 0x7f146805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:31.894 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.893+0000 7f147c98c700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 0x7f146805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f146c009fd0 tx=0x7f146c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:31.897 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:31.897+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1464049be0 con 0x7f14780f7a70 2024-01-18T07:34:32.046 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.045+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 --> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1478108cb0 con 0x7f146805c2a0 2024-01-18T07:34:32.047 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.046+0000 7f14757fa700 1 -- 172.21.15.84:0/735796305 <== mgr.14162 v2:172.21.15.84:6800/2258764086 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27558 (secure 0 0 0) 0x7f1478108cb0 con 0x7f146805c2a0 2024-01-18T07:34:32.047 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 msgr2=0x7f146805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 0x7f146805e760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f146c009fd0 tx=0x7f146c00b040).stop 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 msgr2=0x7f1478107500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1464000c00 tx=0x7f146400b040).stop 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 shutdown_connections 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f146805c2a0 0x7f146805e760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f14780f67d0 0x7f1478107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 --2- 172.21.15.84:0/735796305 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f14780f7a70 0x7f1478107500 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 >> 172.21.15.84:0/735796305 conn(0x7f14780f1f50 msgr2=0x7f14780fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 shutdown_connections 2024-01-18T07:34:32.049 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:32.048+0000 7f147ebf0700 1 -- 172.21.15.84:0/735796305 wait complete. 2024-01-18T07:34:32.050 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-01-18T07:34:32.478 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-01-18T07:34:31.005721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":31112,"kb_used_data":1944,"kb_used_omap":0,"kb_used_meta":29120,"kb_avail":562439800,"statfs":{"total":575970213888,"available":575938355200,"internally_reserved":0,"allocated":1990656,"data_stored":760032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29818880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":"34","reported_epoch":"35","state":"active+clean","last_fresh":"2024-01-18T07:33:52.797273+0000","last_change":"2024-01-18T07:31:42.974429+0000","last_active":"2024-01-18T07:33:52.797273+0000","last_peered":"2024-01-18T07:33:52.797273+0000","last_clean":"2024-01-18T07:33:52.797273+0000","last_became_active":"2024-01-18T07:31:42.974060+0000","last_became_peered":"2024-01-18T07:31:42.974060+0000","last_unstale":"2024-01-18T07:33:52.797273+0000","last_undegraded":"2024-01-18T07:33:52.797273+0000","last_fullsized":"2024-01-18T07:33:52.797273+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T07:28:26.574375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T07:28:26.574375+0000","last_clean_scrub_stamp":"2024-01-18T07:28:26.574375+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5068,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":93740084,"statfs":{"total":95995035648,"available":95989846016,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.621,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.39500000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.66300000000000003,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.65100000000000002,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52600000000000002,"15min":0.53200000000000003},"min":{"1min":0.38400000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.61299999999999999,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.503}]},{"osd":2,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.68100000000000005,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.624,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 18 07:33:44 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.376,"15min":0.378},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.45200000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.437,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5196,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":93739956,"statfs":{"total":95995035648,"available":95989714944,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53900000000000003,"15min":0.55900000000000005},"min":{"1min":0.42699999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.68999999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53900000000000003,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71599999999999997,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.505}]},{"osd":1,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.56499999999999995},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69199999999999995,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53700000000000003,"15min":0.56699999999999995},"min":{"1min":0.435,"5min":0.315,"15min":0.315},"max":{"1min":0.629,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.496}]},{"osd":2,"last update":"Thu Jan 18 07:33:39 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.56799999999999995},"min":{"1min":0.39700000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.64400000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.57199999999999995},"min":{"1min":0.41499999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.73699999999999999,"5min":0.755,"15min":0.755},"last":0.53400000000000003}]},{"osd":4,"last update":"Thu Jan 18 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.41499999999999998,"15min":0.42399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.499,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.52500000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":2,"up_from":17,"seq":73014444152,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.34100000000000003},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.504,"5min":0.504,"15min":0.51000000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.35899999999999999},"min":{"1min":0.249,"5min":0.23799999999999999,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.497},"last":0.23000000000000001}]},{"osd":1,"last update":"Thu Jan 18 07:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.36299999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24199999999999999},"max":{"1min":0.46500000000000002,"5min":0.504,"15min":0.51300000000000001},"last":0.223},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35499999999999998,"15min":0.373},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.51800000000000002,"15min":0.54700000000000004},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Jan 18 07:33:38 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.63400000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.66800000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.439}]},{"osd":4,"last update":"Thu Jan 18 07:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.67500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.71499999999999997,"5min":0.747,"15min":0.747},"last":0.42899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705789,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.20899999999999999},"max":{"1min":0.46100000000000002,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.247},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35199999999999998,"15min":0.34699999999999998},"min":{"1min":0.26400000000000001,"5min":0.252,"15min":0.23000000000000001},"max":{"1min":0.51200000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.28999999999999998}]},{"osd":2,"last update":"Thu Jan 18 07:33:56 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.44800000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.255},{"interface":"front","average":{"1min":0.371,"5min":0.36799999999999999,"15min":0.377},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":0.498,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27800000000000002}]},{"osd":3,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.48499999999999999,"15min":0.48799999999999999},"min":{"1min":0.38800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.60299999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70299999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 18 07:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.69999999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.43099999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65300000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5260,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739892,"statfs":{"total":95995035648,"available":95989649408,"internally_reserved":0,"allocated":331776,"data_stored":126672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 18 07:33:48 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.35399999999999998,"15min":0.33100000000000002},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.35699999999999998,"15min":0.33800000000000002},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.34699999999999998}]},{"osd":2,"last update":"Thu Jan 18 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.38600000000000001,"15min":0.36199999999999999},"min":{"1min":0.254,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.35999999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313}]},{"osd":3,"last update":"Thu Jan 18 07:33:41 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.35999999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.66900000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.63200000000000001}]},{"osd":4,"last update":"Thu Jan 18 07:33:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-18T07:34:32.479 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-01-18T07:34:32.479 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-18T07:34:32.479 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-01-18T07:34:32.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:32.698 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:33.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:32.046621+0000 mgr.smithi084.sihbqg (mgr.14162) 508 : audit [DBG] from='client.14522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T07:34:33.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:33.006502+0000 mgr.smithi084.sihbqg (mgr.14162) 509 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:32.046621+0000 mgr.smithi084.sihbqg (mgr.14162) 508 : audit [DBG] from='client.14522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-18T07:34:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:33.006502+0000 mgr.smithi084.sihbqg (mgr.14162) 509 : cluster [DBG] pgmap v475: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- 172.21.15.84:0/2849517504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 msgr2=0x7f93d00f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 --2- 172.21.15.84:0/2849517504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d00f5930 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f93c0004660 tx=0x7f93c0009b30).stop 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- 172.21.15.84:0/2849517504 shutdown_connections 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 --2- 172.21.15.84:0/2849517504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d00f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 --2- 172.21.15.84:0/2849517504 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d00f5930 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- 172.21.15.84:0/2849517504 >> 172.21.15.84:0/2849517504 conn(0x7f93d00f0c70 msgr2=0x7f93d00f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- 172.21.15.84:0/2849517504 shutdown_connections 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- 172.21.15.84:0/2849517504 wait complete. 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 Processor -- start 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- start start 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:33.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d010c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0060920 con 0x7f93d00f5510 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93d5456700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0060a60 con 0x7f93d00f67b0 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93cffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.692+0000 7f93cffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:33404/0 (socket says 172.21.15.84:33404) 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cffff700 1 -- 172.21.15.84:0/331257223 learned_addr learned my addr 172.21.15.84:0/331257223 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93c7fff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d010c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cffff700 1 -- 172.21.15.84:0/331257223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 msgr2=0x7f93d010c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cffff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d010c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cffff700 1 -- 172.21.15.84:0/331257223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c0005040 con 0x7f93d00f5510 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93c7fff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d010c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cffff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f93c0009f50 tx=0x7f93c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c0018070 con 0x7f93d00f5510 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d010c670 con 0x7f93d00f5510 2024-01-18T07:34:33.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93c0013410 con 0x7f93d00f5510 2024-01-18T07:34:33.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93c001c620 con 0x7f93d00f5510 2024-01-18T07:34:33.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.693+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d010cba0 con 0x7f93d00f5510 2024-01-18T07:34:33.696 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.694+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d0059100 con 0x7f93d00f5510 2024-01-18T07:34:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.694+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f93c001d420 con 0x7f93d00f5510 2024-01-18T07:34:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.695+0000 7f93cd7fa700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 0x7f93bc05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.695+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f93c0077a20 con 0x7f93d00f5510 2024-01-18T07:34:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.695+0000 7f93c7fff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 0x7f93bc05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:33.697 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.695+0000 7f93c7fff700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 0x7f93bc05e770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f93b80041d0 tx=0x7f93b8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:33.699 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.698+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c004aa20 con 0x7f93d00f5510 2024-01-18T07:34:33.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.904+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f93d0045ad0 con 0x7f93d00f5510 2024-01-18T07:34:33.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.905+0000 7f93cd7fa700 1 -- 172.21.15.84:0/331257223 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f93c004e040 con 0x7f93d00f5510 2024-01-18T07:34:33.905 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 msgr2=0x7f93bc05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 0x7f93bc05e770 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f93b80041d0 tx=0x7f93b8009040).stop 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 msgr2=0x7f93d0109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f93c0009f50 tx=0x7f93c0009b30).stop 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 shutdown_connections 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f93d00f5510 0x7f93d0109c20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f93bc05c2b0 0x7f93bc05e770 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 --2- 172.21.15.84:0/331257223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93d00f67b0 0x7f93d010c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 >> 172.21.15.84:0/331257223 conn(0x7f93d00f0c70 msgr2=0x7f93d00fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 shutdown_connections 2024-01-18T07:34:33.907 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:33.906+0000 7f93d5456700 1 -- 172.21.15.84:0/331257223 wait complete. 2024-01-18T07:34:34.252 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:33.905792+0000 mon.smithi084 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.84:0/331257223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:34.334 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:33.905792+0000 mon.smithi084 (mon.0) 502 : audit [DBG] from='client.? 172.21.15.84:0/331257223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:35.006866+0000 mgr.smithi084.sihbqg (mgr.14162) 510 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:35.051854+0000 mon.smithi084 (mon.0) 503 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:34:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:35.061941+0000 mon.smithi084 (mon.0) 504 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:34:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:35.006866+0000 mgr.smithi084.sihbqg (mgr.14162) 510 : cluster [DBG] pgmap v476: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:35.051854+0000 mon.smithi084 (mon.0) 503 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:34:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:35.061941+0000 mon.smithi084 (mon.0) 504 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:34:37.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:37.366 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:37.007320+0000 mgr.smithi084.sihbqg (mgr.14162) 511 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:37.007320+0000 mgr.smithi084.sihbqg (mgr.14162) 511 : cluster [DBG] pgmap v477: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:37.567 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- 172.21.15.84:0/2429860948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c060870 msgr2=0x7f758c0f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 --2- 172.21.15.84:0/2429860948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c060870 0x7f758c0f91f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7574004660 tx=0x7f7574009b30).stop 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- 172.21.15.84:0/2429860948 shutdown_connections 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 --2- 172.21.15.84:0/2429860948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c0f9770 0x7f758c0fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 --2- 172.21.15.84:0/2429860948 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c060870 0x7f758c0f91f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- 172.21.15.84:0/2429860948 >> 172.21.15.84:0/2429860948 conn(0x7f758c0f0c70 msgr2=0x7f758c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- 172.21.15.84:0/2429860948 shutdown_connections 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- 172.21.15.84:0/2429860948 wait complete. 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 Processor -- start 2024-01-18T07:34:38.471 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- start start 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c060870 0x7f758c0f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0f56e0 con 0x7f758c0f9770 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.470+0000 7f75909b6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0f5820 con 0x7f758c060870 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45664/0 (socket says 172.21.15.84:45664) 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 -- 172.21.15.84:0/2509525871 learned_addr learned my addr 172.21.15.84:0/2509525871 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758b7fe700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c060870 0x7f758c0f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 -- 172.21.15.84:0/2509525871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c060870 msgr2=0x7f758c0f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c060870 0x7f758c0f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:38.472 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 -- 172.21.15.84:0/2509525871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7574005040 con 0x7f758c0f9770 2024-01-18T07:34:38.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f758affd700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f757c0093f0 tx=0x7f757c006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:38.473 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f757c013640 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c0f5b00 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f757c013c50 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f757c012c80 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.471+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c10ef30 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.472+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c045ad0 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.472+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f757c01a060 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.473+0000 7f7583fff700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 0x7f757805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.473+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f757c075ba0 con 0x7f758c0f9770 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.473+0000 7f758b7fe700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 0x7f757805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:38.474 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.473+0000 7f758b7fe700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 0x7f757805e770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7574004630 tx=0x7f7574009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:38.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.476+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f757c048ba0 con 0x7f758c0f9770 2024-01-18T07:34:38.684 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.684+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f758c027a00 con 0x7f758c0f9770 2024-01-18T07:34:38.685 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.684+0000 7f7583fff700 1 -- 172.21.15.84:0/2509525871 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f757c04c1c0 con 0x7f758c0f9770 2024-01-18T07:34:38.685 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:38.686 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 msgr2=0x7f757805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 0x7f757805e770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7574004630 tx=0x7f7574009b30).stop 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 msgr2=0x7f758c0f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f757c0093f0 tx=0x7f757c006ec0).stop 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 shutdown_connections 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f757805c2b0 0x7f757805e770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f758c060870 0x7f758c0f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 --2- 172.21.15.84:0/2509525871 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f758c0f9770 0x7f758c0f50f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 >> 172.21.15.84:0/2509525871 conn(0x7f758c0f0c70 msgr2=0x7f758c0f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 shutdown_connections 2024-01-18T07:34:38.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:38.686+0000 7f75909b6700 1 -- 172.21.15.84:0/2509525871 wait complete. 2024-01-18T07:34:38.992 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:38.685559+0000 mon.smithi084 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.84:0/2509525871' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:39.052 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:39.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:38.685559+0000 mon.smithi084 (mon.0) 505 : audit [DBG] from='client.? 172.21.15.84:0/2509525871' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:40.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:39.007649+0000 mgr.smithi084.sihbqg (mgr.14162) 512 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:40.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:39.007649+0000 mgr.smithi084.sihbqg (mgr.14162) 512 : cluster [DBG] pgmap v478: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:41.007920+0000 mgr.smithi084.sihbqg (mgr.14162) 513 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:41.007920+0000 mgr.smithi084.sihbqg (mgr.14162) 513 : cluster [DBG] pgmap v479: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:42.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:42.288 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- 172.21.15.84:0/1398973060 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 msgr2=0x7f1de40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 --2- 172.21.15.84:0/1398973060 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f5ae0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f1dcc004660 tx=0x7f1dcc009b30).stop 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- 172.21.15.84:0/1398973060 shutdown_connections 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 --2- 172.21.15.84:0/1398973060 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 0x7f1de40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 --2- 172.21.15.84:0/1398973060 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f5ae0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- 172.21.15.84:0/1398973060 >> 172.21.15.84:0/1398973060 conn(0x7f1de40f0e20 msgr2=0x7f1de40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- 172.21.15.84:0/1398973060 shutdown_connections 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- 172.21.15.84:0/1398973060 wait complete. 2024-01-18T07:34:43.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 Processor -- start 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.241+0000 7f1de916b700 1 -- start start 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 0x7f1de40f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40f8290 con 0x7f1de40f56c0 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40f83d0 con 0x7f1de40f6960 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45678/0 (socket says 172.21.15.84:45678) 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 -- 172.21.15.84:0/2549961458 learned_addr learned my addr 172.21.15.84:0/2549961458 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de259c700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 0x7f1de40f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 -- 172.21.15.84:0/2549961458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 msgr2=0x7f1de40f7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 0x7f1de40f7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 -- 172.21.15.84:0/2549961458 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc005040 con 0x7f1de40f56c0 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de2d9d700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f1dcc009f50 tx=0x7f1dcc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:43.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dcc018070 con 0x7f1de40f56c0 2024-01-18T07:34:43.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dcc013410 con 0x7f1de40f56c0 2024-01-18T07:34:43.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de40f8650 con 0x7f1de40f56c0 2024-01-18T07:34:43.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1dcc01c620 con 0x7f1de40f56c0 2024-01-18T07:34:43.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.242+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de410f110 con 0x7f1de40f56c0 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.243+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de4045ad0 con 0x7f1de40f56c0 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.244+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1dcc01d420 con 0x7f1de40f56c0 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.244+0000 7f1ddb7fe700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 0x7f1dd005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.244+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1dcc0779f0 con 0x7f1de40f56c0 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.244+0000 7f1de259c700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 0x7f1dd005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:43.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.244+0000 7f1de259c700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 0x7f1dd005e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f1dd400b830 tx=0x7f1dd4008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:43.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.248+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dcc079e60 con 0x7f1de40f56c0 2024-01-18T07:34:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:43.008269+0000 mgr.smithi084.sihbqg (mgr.14162) 514 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:43.008269+0000 mgr.smithi084.sihbqg (mgr.14162) 514 : cluster [DBG] pgmap v480: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:43.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.456+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1de40fded0 con 0x7f1de40f56c0 2024-01-18T07:34:43.456 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.456+0000 7f1ddb7fe700 1 -- 172.21.15.84:0/2549961458 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f1dcc04a9f0 con 0x7f1de40f56c0 2024-01-18T07:34:43.456 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.457+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 msgr2=0x7f1dd005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.457+0000 7f1de916b700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 0x7f1dd005e760 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f1dd400b830 tx=0x7f1dd4008040).stop 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.457+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 msgr2=0x7f1de40f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.457+0000 7f1de916b700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f1dcc009f50 tx=0x7f1dcc009b30).stop 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 shutdown_connections 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1de40f56c0 0x7f1de40f9820 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1dd005c2a0 0x7f1dd005e760 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 --2- 172.21.15.84:0/2549961458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1de40f6960 0x7f1de40f7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:43.458 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 >> 172.21.15.84:0/2549961458 conn(0x7f1de40f0e20 msgr2=0x7f1de40603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:43.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 shutdown_connections 2024-01-18T07:34:43.459 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:43.458+0000 7f1de916b700 1 -- 172.21.15.84:0/2549961458 wait complete. 2024-01-18T07:34:43.874 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:44.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:43.457049+0000 mon.smithi084 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.84:0/2549961458' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:44.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:43.457049+0000 mon.smithi084 (mon.0) 506 : audit [DBG] from='client.? 172.21.15.84:0/2549961458' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:45.008685+0000 mgr.smithi084.sihbqg (mgr.14162) 515 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:45.008685+0000 mgr.smithi084.sihbqg (mgr.14162) 515 : cluster [DBG] pgmap v481: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:46.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:47.112 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:47.261 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:47.009067+0000 mgr.smithi084.sihbqg (mgr.14162) 516 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:47.009067+0000 mgr.smithi084.sihbqg (mgr.14162) 516 : cluster [DBG] pgmap v482: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 -- 172.21.15.84:0/228038819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f62f0 msgr2=0x7fa4380f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 --2- 172.21.15.84:0/228038819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f62f0 0x7fa4380f6750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fa428004660 tx=0x7fa428009b30).stop 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 -- 172.21.15.84:0/228038819 shutdown_connections 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 --2- 172.21.15.84:0/228038819 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f62f0 0x7fa4380f6750 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 --2- 172.21.15.84:0/228038819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f7ce0 0x7fa4380f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 -- 172.21.15.84:0/228038819 >> 172.21.15.84:0/228038819 conn(0x7fa4380f1d70 msgr2=0x7fa4380f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 -- 172.21.15.84:0/228038819 shutdown_connections 2024-01-18T07:34:48.118 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.115+0000 7fa43ef41700 1 -- 172.21.15.84:0/228038819 wait complete. 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 Processor -- start 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 -- start start 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 0x7fa438106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438107a40 con 0x7fa4380f7ce0 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43ef41700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa438107b80 con 0x7fa4380f62f0 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45756/0 (socket says 172.21.15.84:45756) 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 -- 172.21.15.84:0/1701363735 learned_addr learned my addr 172.21.15.84:0/1701363735 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43df3f700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 0x7fa438106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 -- 172.21.15.84:0/1701363735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 msgr2=0x7fa438106fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 0x7fa438106fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43d73e700 1 -- 172.21.15.84:0/1701363735 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa428005040 con 0x7fa4380f7ce0 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.116+0000 7fa43df3f700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 0x7fa438106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:34:48.119 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa43d73e700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa428009830 tx=0x7fa428009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa428018070 con 0x7fa4380f7ce0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4381084c0 con 0x7fa4380f7ce0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa428013410 con 0x7fa4380f7ce0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa42801c620 con 0x7fa4380f7ce0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.117+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa438108a20 con 0x7fa4380f7ce0 2024-01-18T07:34:48.120 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.118+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa438045ad0 con 0x7fa4380f7ce0 2024-01-18T07:34:48.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.118+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa42801c8e0 con 0x7fa4380f7ce0 2024-01-18T07:34:48.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.119+0000 7fa42e7fc700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 0x7fa42405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:48.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.119+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa428077dd0 con 0x7fa4380f7ce0 2024-01-18T07:34:48.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.119+0000 7fa43df3f700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 0x7fa42405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:48.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.119+0000 7fa43df3f700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 0x7fa42405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa4380f7730 tx=0x7fa434009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:48.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.122+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa42804ae60 con 0x7fa4380f7ce0 2024-01-18T07:34:48.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.328+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa438027a00 con 0x7fa4380f7ce0 2024-01-18T07:34:48.329 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.329+0000 7fa42e7fc700 1 -- 172.21.15.84:0/1701363735 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa42804e480 con 0x7fa4380f7ce0 2024-01-18T07:34:48.329 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 msgr2=0x7fa42405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 0x7fa42405e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa4380f7730 tx=0x7fa434009040).stop 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 msgr2=0x7fa438107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa428009830 tx=0x7fa428009b30).stop 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 shutdown_connections 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa42405c2b0 0x7fa42405e770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4380f62f0 0x7fa438106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 --2- 172.21.15.84:0/1701363735 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa4380f7ce0 0x7fa438107490 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 >> 172.21.15.84:0/1701363735 conn(0x7fa4380f1d70 msgr2=0x7fa4380fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.330+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 shutdown_connections 2024-01-18T07:34:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:48.331+0000 7fa43ef41700 1 -- 172.21.15.84:0/1701363735 wait complete. 2024-01-18T07:34:48.631 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:48.329770+0000 mon.smithi084 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.84:0/1701363735' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:48.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:48.329770+0000 mon.smithi084 (mon.0) 507 : audit [DBG] from='client.? 172.21.15.84:0/1701363735' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:48.700 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:49.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:49.009430+0000 mgr.smithi084.sihbqg (mgr.14162) 517 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:49.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:49.009430+0000 mgr.smithi084.sihbqg (mgr.14162) 517 : cluster [DBG] pgmap v483: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:51.009941+0000 mgr.smithi084.sihbqg (mgr.14162) 518 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:51.009941+0000 mgr.smithi084.sihbqg (mgr.14162) 518 : cluster [DBG] pgmap v484: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:51.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:51.935 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- 172.21.15.84:0/1432778455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 msgr2=0x7f96700f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 --2- 172.21.15.84:0/1432778455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96700f6be0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9664004660 tx=0x7f9664009b30).stop 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- 172.21.15.84:0/1432778455 shutdown_connections 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 --2- 172.21.15.84:0/1432778455 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 0x7f96700f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 --2- 172.21.15.84:0/1432778455 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96700f6be0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- 172.21.15.84:0/1432778455 >> 172.21.15.84:0/1432778455 conn(0x7f96700f1f20 msgr2=0x7f96700f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- 172.21.15.84:0/1432778455 shutdown_connections 2024-01-18T07:34:52.890 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- 172.21.15.84:0/1432778455 wait complete. 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 Processor -- start 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.887+0000 7f967621c700 1 -- start start 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 0x7f9670107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9670107b40 con 0x7f96700f67c0 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9670107c80 con 0x7f96700f7a60 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:45788/0 (socket says 172.21.15.84:45788) 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 -- 172.21.15.84:0/3989081766 learned_addr learned my addr 172.21.15.84:0/3989081766 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966effd700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 0x7f9670107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 -- 172.21.15.84:0/3989081766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 msgr2=0x7f9670107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 0x7f9670107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 -- 172.21.15.84:0/3989081766 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9664005040 con 0x7f96700f67c0 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f966f7fe700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9664004e30 tx=0x7f9664009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:52.891 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9664018070 con 0x7f96700f67c0 2024-01-18T07:34:52.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96701085c0 con 0x7f96700f67c0 2024-01-18T07:34:52.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9664013410 con 0x7f96700f67c0 2024-01-18T07:34:52.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f966401de20 con 0x7f96700f67c0 2024-01-18T07:34:52.892 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.888+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9670108b20 con 0x7f96700f67c0 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.889+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9670045ad0 con 0x7f96700f67c0 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.890+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f966401d420 con 0x7f96700f67c0 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.890+0000 7f9657fff700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 0x7f965805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.890+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f96640777b0 con 0x7f96700f67c0 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.890+0000 7f966effd700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 0x7f965805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:52.893 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.891+0000 7f966effd700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 0x7f965805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9660004c40 tx=0x7f966000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:52.895 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:52.894+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f966404a7b0 con 0x7f96700f67c0 2024-01-18T07:34:53.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.103+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9670027a00 con 0x7f96700f67c0 2024-01-18T07:34:53.104 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.103+0000 7f9657fff700 1 -- 172.21.15.84:0/3989081766 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f966404ddd0 con 0x7f96700f67c0 2024-01-18T07:34:53.104 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 msgr2=0x7f965805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 0x7f965805e760 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9660004c40 tx=0x7f966000b040).stop 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 msgr2=0x7f96701070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9664004e30 tx=0x7f9664009b30).stop 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 shutdown_connections 2024-01-18T07:34:53.106 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f96700f67c0 0x7f96701070a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f965805c2a0 0x7f965805e760 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 --2- 172.21.15.84:0/3989081766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f96700f7a60 0x7f9670107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 >> 172.21.15.84:0/3989081766 conn(0x7f96700f1f20 msgr2=0x7f96700fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 shutdown_connections 2024-01-18T07:34:53.107 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:53.105+0000 7f967621c700 1 -- 172.21.15.84:0/3989081766 wait complete. 2024-01-18T07:34:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:53.010291+0000 mgr.smithi084.sihbqg (mgr.14162) 519 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:53.010291+0000 mgr.smithi084.sihbqg (mgr.14162) 519 : cluster [DBG] pgmap v485: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:53.584 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:54.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:53.105018+0000 mon.smithi084 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.84:0/3989081766' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:54.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:53.105018+0000 mon.smithi084 (mon.0) 508 : audit [DBG] from='client.? 172.21.15.84:0/3989081766' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:55.010831+0000 mgr.smithi084.sihbqg (mgr.14162) 520 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:55.010831+0000 mgr.smithi084.sihbqg (mgr.14162) 520 : cluster [DBG] pgmap v486: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:56.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:34:56.821 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:34:57.277 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:57.011261+0000 mgr.smithi084.sihbqg (mgr.14162) 521 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:57.011261+0000 mgr.smithi084.sihbqg (mgr.14162) 521 : cluster [DBG] pgmap v487: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/677278811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f6610 msgr2=0x7f2a480f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/677278811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f6610 0x7f2a480f6a30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f2a44004660 tx=0x7f2a44009b30).stop 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/677278811 shutdown_connections 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/677278811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f78b0 0x7f2a480f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/677278811 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f6610 0x7f2a480f6a30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:57.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/677278811 >> 172.21.15.84:0/677278811 conn(0x7f2a480f1d70 msgr2=0x7f2a480f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/677278811 shutdown_connections 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/677278811 wait complete. 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 Processor -- start 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- start start 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f6610 0x7f2a48106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a48107980 con 0x7f2a480f78b0 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4e9aa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a48107ac0 con 0x7f2a480f6610 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.812+0000 7f2a4d1a7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:60664/0 (socket says 172.21.15.84:60664) 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 -- 172.21.15.84:0/4058455195 learned_addr learned my addr 172.21.15.84:0/4058455195 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d9a8700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f6610 0x7f2a48106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:57.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 -- 172.21.15.84:0/4058455195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f6610 msgr2=0x7f2a48106ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f6610 0x7f2a48106ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 -- 172.21.15.84:0/4058455195 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44005040 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4d1a7700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2a3800c530 tx=0x7f2a38007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a3800f650 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a3800fc60 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a48108460 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a38012c80 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.813+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a48108a20 con 0x7f2a480f78b0 2024-01-18T07:34:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.814+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a48045ad0 con 0x7f2a480f78b0 2024-01-18T07:34:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.814+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2a38010070 con 0x7f2a480f78b0 2024-01-18T07:34:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.815+0000 7f2a3e7fc700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 0x7f2a3405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:34:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.815+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2a380758c0 con 0x7f2a480f78b0 2024-01-18T07:34:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.815+0000 7f2a4d9a8700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 0x7f2a3405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:34:57.818 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.815+0000 7f2a4d9a8700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 0x7f2a3405e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2a44004e30 tx=0x7f2a44009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:34:57.820 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:57.819+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a380452b0 con 0x7f2a480f78b0 2024-01-18T07:34:58.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.026+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2a48027a00 con 0x7f2a480f78b0 2024-01-18T07:34:58.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.027+0000 7f2a3e7fc700 1 -- 172.21.15.84:0/4058455195 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f2a3804c2f0 con 0x7f2a480f78b0 2024-01-18T07:34:58.028 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 msgr2=0x7f2a3405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 0x7f2a3405e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2a44004e30 tx=0x7f2a44009b30).stop 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 msgr2=0x7f2a481073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2a3800c530 tx=0x7f2a38007e20).stop 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 shutdown_connections 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2a3405c2b0 0x7f2a3405e770 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a480f6610 0x7f2a48106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 --2- 172.21.15.84:0/4058455195 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2a480f78b0 0x7f2a481073d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 >> 172.21.15.84:0/4058455195 conn(0x7f2a480f1d70 msgr2=0x7f2a480fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 shutdown_connections 2024-01-18T07:34:58.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:34:58.028+0000 7f2a4e9aa700 1 -- 172.21.15.84:0/4058455195 wait complete. 2024-01-18T07:34:58.371 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:34:58.028758+0000 mon.smithi084 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.84:0/4058455195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:58.414 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:34:58.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:34:58.028758+0000 mon.smithi084 (mon.0) 509 : audit [DBG] from='client.? 172.21.15.84:0/4058455195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:34:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:34:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:34:59.011600+0000 mgr.smithi084.sihbqg (mgr.14162) 522 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:34:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:34:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:34:59.011600+0000 mgr.smithi084.sihbqg (mgr.14162) 522 : cluster [DBG] pgmap v488: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:01.011926+0000 mgr.smithi084.sihbqg (mgr.14162) 523 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:01.011926+0000 mgr.smithi084.sihbqg (mgr.14162) 523 : cluster [DBG] pgmap v489: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:01.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:01.647 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:02.635 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/1231318947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 msgr2=0x7ff2ec0fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:02.635 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/1231318947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec0fbd80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff2e00045a0 tx=0x7ff2e0009b30).stop 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/1231318947 shutdown_connections 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/1231318947 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec0fbd80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/1231318947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 0x7ff2ec0f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/1231318947 >> 172.21.15.84:0/1231318947 conn(0x7ff2ec0f0e20 msgr2=0x7ff2ec0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/1231318947 shutdown_connections 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.632+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/1231318947 wait complete. 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 Processor -- start 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 -- start start 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 0x7ff2ec107020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec107ac0 con 0x7ff2ec0f9930 2024-01-18T07:35:02.636 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2f1e9f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec107c00 con 0x7ff2ec0607d0 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2eaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:60674/0 (socket says 172.21.15.84:60674) 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2eaffd700 1 -- 172.21.15.84:0/727361761 learned_addr learned my addr 172.21.15.84:0/727361761 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.633+0000 7ff2eb7fe700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 0x7ff2ec107020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2eaffd700 1 -- 172.21.15.84:0/727361761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 msgr2=0x7ff2ec107020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2eaffd700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 0x7ff2ec107020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2eaffd700 1 -- 172.21.15.84:0/727361761 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0005040 con 0x7ff2ec0f9930 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2eaffd700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff2e0000c00 tx=0x7ff2e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2e000e070 con 0x7ff2ec0f9930 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2e0007ba0 con 0x7ff2ec0f9930 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2e0019070 con 0x7ff2ec0f9930 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ec108540 con 0x7ff2ec0f9930 2024-01-18T07:35:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.634+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ec1089d0 con 0x7ff2ec0f9930 2024-01-18T07:35:02.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.635+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ec045ad0 con 0x7ff2ec0f9930 2024-01-18T07:35:02.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.635+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff2e0014420 con 0x7ff2ec0f9930 2024-01-18T07:35:02.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.636+0000 7ff2d3fff700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 0x7ff2d405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:02.638 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.636+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff2e0076f00 con 0x7ff2ec0f9930 2024-01-18T07:35:02.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.636+0000 7ff2eb7fe700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 0x7ff2d405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:02.639 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.636+0000 7ff2eb7fe700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 0x7ff2d405e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff2dc009880 tx=0x7ff2dc00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:02.641 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.640+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2e0049f00 con 0x7ff2ec0f9930 2024-01-18T07:35:02.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.846+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff2ec0fef60 con 0x7ff2ec0f9930 2024-01-18T07:35:02.848 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.847+0000 7ff2d3fff700 1 -- 172.21.15.84:0/727361761 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff2e004d520 con 0x7ff2ec0f9930 2024-01-18T07:35:02.848 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 msgr2=0x7ff2d405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 0x7ff2d405e760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ff2dc009880 tx=0x7ff2dc00b040).stop 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 msgr2=0x7ff2ec107510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff2e0000c00 tx=0x7ff2e000b040).stop 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 shutdown_connections 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff2d405c2a0 0x7ff2d405e760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff2ec0607d0 0x7ff2ec107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 --2- 172.21.15.84:0/727361761 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff2ec0f9930 0x7ff2ec107510 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 >> 172.21.15.84:0/727361761 conn(0x7ff2ec0f0e20 msgr2=0x7ff2ec0f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 shutdown_connections 2024-01-18T07:35:02.850 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:02.848+0000 7ff2f1e9f700 1 -- 172.21.15.84:0/727361761 wait complete. 2024-01-18T07:35:03.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:02.848821+0000 mon.smithi084 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.84:0/727361761' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:03.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:02.848821+0000 mon.smithi084 (mon.0) 510 : audit [DBG] from='client.? 172.21.15.84:0/727361761' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:03.282 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:04.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:03.012273+0000 mgr.smithi084.sihbqg (mgr.14162) 524 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:04.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:03.012273+0000 mgr.smithi084.sihbqg (mgr.14162) 524 : cluster [DBG] pgmap v490: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:05.012609+0000 mgr.smithi084.sihbqg (mgr.14162) 525 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:05.012609+0000 mgr.smithi084.sihbqg (mgr.14162) 525 : cluster [DBG] pgmap v491: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:06.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:06.518 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:07.254 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:07.013094+0000 mgr.smithi084.sihbqg (mgr.14162) 526 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:07.413 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.411+0000 7fc667ee3700 1 -- 172.21.15.84:0/3790837288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f7a60 msgr2=0x7fc6600f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.411+0000 7fc667ee3700 1 --2- 172.21.15.84:0/3790837288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f7a60 0x7fc6600f7ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fc650004660 tx=0x7fc650009b30).stop 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- 172.21.15.84:0/3790837288 shutdown_connections 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 --2- 172.21.15.84:0/3790837288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f7a60 0x7fc6600f7ec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 --2- 172.21.15.84:0/3790837288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f67c0 0x7fc6600f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- 172.21.15.84:0/3790837288 >> 172.21.15.84:0/3790837288 conn(0x7fc6600f1f20 msgr2=0x7fc6600f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- 172.21.15.84:0/3790837288 shutdown_connections 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- 172.21.15.84:0/3790837288 wait complete. 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 Processor -- start 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- start start 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f7a60 0x7fc6601075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc660107b50 con 0x7fc6600f67c0 2024-01-18T07:35:07.414 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc667ee3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc660107c90 con 0x7fc6600f7a60 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc665c7f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc665c7f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36144/0 (socket says 172.21.15.84:36144) 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.412+0000 7fc665c7f700 1 -- 172.21.15.84:0/116192037 learned_addr learned my addr 172.21.15.84:0/116192037 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc66547e700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f7a60 0x7fc6601075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc665c7f700 1 -- 172.21.15.84:0/116192037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f7a60 msgr2=0x7fc6601075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc665c7f700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f7a60 0x7fc6601075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc665c7f700 1 -- 172.21.15.84:0/116192037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650005040 con 0x7fc6600f67c0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc665c7f700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fc65c002700 tx=0x7fc65c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc65c0079c0 con 0x7fc6600f67c0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc65c00f040 con 0x7fc6600f67c0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc660108630 con 0x7fc6600f67c0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc65c014860 con 0x7fc6600f67c0 2024-01-18T07:35:07.415 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.413+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc660108bf0 con 0x7fc6600f67c0 2024-01-18T07:35:07.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:07.013094+0000 mgr.smithi084.sihbqg (mgr.14162) 526 : cluster [DBG] pgmap v492: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.414+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc660045ad0 con 0x7fc6600f67c0 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.414+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc65c01b060 con 0x7fc6600f67c0 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.415+0000 7fc6567fc700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 0x7fc64c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.415+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc65c0759d0 con 0x7fc6600f67c0 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.415+0000 7fc66547e700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 0x7fc64c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:07.417 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.415+0000 7fc66547e700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 0x7fc64c05e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc650009f50 tx=0x7fc650009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:07.419 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.419+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc65c0489d0 con 0x7fc6600f67c0 2024-01-18T07:35:07.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.627+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6600ff0d0 con 0x7fc6600f67c0 2024-01-18T07:35:07.628 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.628+0000 7fc6567fc700 1 -- 172.21.15.84:0/116192037 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc65c04bff0 con 0x7fc6600f67c0 2024-01-18T07:35:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.629+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 msgr2=0x7fc64c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.629+0000 7fc667ee3700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 0x7fc64c05e760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc650009f50 tx=0x7fc650009b30).stop 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.629+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 msgr2=0x7fc6601070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.629+0000 7fc667ee3700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fc65c002700 tx=0x7fc65c009d80).stop 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 shutdown_connections 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc6600f67c0 0x7fc6601070b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc64c05c2a0 0x7fc64c05e760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 --2- 172.21.15.84:0/116192037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6600f7a60 0x7fc6601075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 >> 172.21.15.84:0/116192037 conn(0x7fc6600f1f20 msgr2=0x7fc6600fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 shutdown_connections 2024-01-18T07:35:07.630 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:07.630+0000 7fc667ee3700 1 -- 172.21.15.84:0/116192037 wait complete. 2024-01-18T07:35:08.004 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:08.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:07.628951+0000 mon.smithi084 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.84:0/116192037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:08.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:07.628951+0000 mon.smithi084 (mon.0) 511 : audit [DBG] from='client.? 172.21.15.84:0/116192037' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:09.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:09.013479+0000 mgr.smithi084.sihbqg (mgr.14162) 527 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:09.013479+0000 mgr.smithi084.sihbqg (mgr.14162) 527 : cluster [DBG] pgmap v493: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:11.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:11.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:11.347 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:11.013896+0000 mgr.smithi084.sihbqg (mgr.14162) 528 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:11.013896+0000 mgr.smithi084.sihbqg (mgr.14162) 528 : cluster [DBG] pgmap v494: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 -- 172.21.15.84:0/4031540755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 msgr2=0x7ff28c0f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 --2- 172.21.15.84:0/4031540755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c0f7d10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff27c004660 tx=0x7ff27c009b30).stop 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 -- 172.21.15.84:0/4031540755 shutdown_connections 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 --2- 172.21.15.84:0/4031540755 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c0f7d10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 --2- 172.21.15.84:0/4031540755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c0f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:12.209 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.208+0000 7ff2925bd700 1 -- 172.21.15.84:0/4031540755 >> 172.21.15.84:0/4031540755 conn(0x7ff28c0f1d70 msgr2=0x7ff28c0f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:12.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.209+0000 7ff2925bd700 1 -- 172.21.15.84:0/4031540755 shutdown_connections 2024-01-18T07:35:12.210 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.209+0000 7ff2925bd700 1 -- 172.21.15.84:0/4031540755 wait complete. 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.209+0000 7ff2925bd700 1 Processor -- start 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.209+0000 7ff2925bd700 1 -- start start 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c106f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c107a70 con 0x7ff28c0f78b0 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c108320 con 0x7ff28c0f6610 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:36164/0 (socket says 172.21.15.84:36164) 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 -- 172.21.15.84:0/1793031300 learned_addr learned my addr 172.21.15.84:0/1793031300 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2915bb700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:12.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 -- 172.21.15.84:0/1793031300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 msgr2=0x7ff28c106f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c106f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 -- 172.21.15.84:0/1793031300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c005040 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2915bb700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff290dba700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff27c009830 tx=0x7ff27c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c018070 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c1085a0 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff27c013410 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff27c01c620 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.210+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c108b00 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.211+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c059100 con 0x7ff28c0f78b0 2024-01-18T07:35:12.213 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.212+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff27c01c8e0 con 0x7ff28c0f78b0 2024-01-18T07:35:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.212+0000 7ff281ffb700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 0x7ff27805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.212+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff27c077a60 con 0x7ff28c0f78b0 2024-01-18T07:35:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.212+0000 7ff2915bb700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 0x7ff27805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:12.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.213+0000 7ff2915bb700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 0x7ff27805e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff28c0f7710 tx=0x7ff288008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.216+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff27c04aa00 con 0x7ff28c0f78b0 2024-01-18T07:35:12.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.422+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff28c0fee40 con 0x7ff28c0f78b0 2024-01-18T07:35:12.423 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.423+0000 7ff281ffb700 1 -- 172.21.15.84:0/1793031300 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff27c04e020 con 0x7ff28c0f78b0 2024-01-18T07:35:12.423 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.424+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 msgr2=0x7ff27805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.424+0000 7ff2925bd700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 0x7ff27805e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff28c0f7710 tx=0x7ff288008040).stop 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.424+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 msgr2=0x7ff28c107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.424+0000 7ff2925bd700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7ff27c009830 tx=0x7ff27c009b30).stop 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 shutdown_connections 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff27805c2b0 0x7ff27805e770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:12.425 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff28c0f6610 0x7ff28c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:12.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 --2- 172.21.15.84:0/1793031300 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff28c0f78b0 0x7ff28c107430 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:12.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 >> 172.21.15.84:0/1793031300 conn(0x7ff28c0f1d70 msgr2=0x7ff28c0fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:12.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 shutdown_connections 2024-01-18T07:35:12.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:12.425+0000 7ff2925bd700 1 -- 172.21.15.84:0/1793031300 wait complete. 2024-01-18T07:35:12.735 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:12.423901+0000 mon.smithi084 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.84:0/1793031300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:12.836 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:12.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:12.423901+0000 mon.smithi084 (mon.0) 512 : audit [DBG] from='client.? 172.21.15.84:0/1793031300' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:13.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:13.014208+0000 mgr.smithi084.sihbqg (mgr.14162) 529 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:13.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:13.014208+0000 mgr.smithi084.sihbqg (mgr.14162) 529 : cluster [DBG] pgmap v495: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:15.014606+0000 mgr.smithi084.sihbqg (mgr.14162) 530 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:15.014606+0000 mgr.smithi084.sihbqg (mgr.14162) 530 : cluster [DBG] pgmap v496: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:15.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:16.073 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 -- 172.21.15.84:0/67089974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f6960 msgr2=0x7f34f00f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 --2- 172.21.15.84:0/67089974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f6960 0x7f34f00f6dc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f34ec004660 tx=0x7f34ec009b30).stop 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 -- 172.21.15.84:0/67089974 shutdown_connections 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 --2- 172.21.15.84:0/67089974 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f6960 0x7f34f00f6dc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 --2- 172.21.15.84:0/67089974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f56c0 0x7f34f00f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 -- 172.21.15.84:0/67089974 >> 172.21.15.84:0/67089974 conn(0x7f34f00f0e20 msgr2=0x7f34f00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 -- 172.21.15.84:0/67089974 shutdown_connections 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.118+0000 7f34f81c9700 1 -- 172.21.15.84:0/67089974 wait complete. 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 Processor -- start 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 -- start start 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f6960 0x7f34f010c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:17.121 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00609b0 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0060af0 con 0x7f34f00f6960 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55908/0 (socket says 172.21.15.84:55908) 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 -- 172.21.15.84:0/3478662601 learned_addr learned my addr 172.21.15.84:0/3478662601 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5764700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f6960 0x7f34f010c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 -- 172.21.15.84:0/3478662601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f6960 msgr2=0x7f34f010c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f6960 0x7f34f010c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 -- 172.21.15.84:0/3478662601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec005040 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f5f65700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f34e00093f0 tx=0x7f34e0006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e0013640 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e0009e40 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f010c900 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.119+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e000dbf0 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.120+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f010cec0 con 0x7f34f00f56c0 2024-01-18T07:35:17.122 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.120+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0045ad0 con 0x7f34f00f56c0 2024-01-18T07:35:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.121+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f34e001a060 con 0x7f34f00f56c0 2024-01-18T07:35:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.121+0000 7f34e67fc700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 0x7f34dc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.121+0000 7f34f5764700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 0x7f34dc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.121+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f34e0075c10 con 0x7f34f00f56c0 2024-01-18T07:35:17.123 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.122+0000 7f34f5764700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 0x7f34dc05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f34ec009d60 tx=0x7f34ec009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:17.125 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.125+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e0048bb0 con 0x7f34f00f56c0 2024-01-18T07:35:17.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.330+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f34f0027a00 con 0x7f34f00f56c0 2024-01-18T07:35:17.330 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.330+0000 7f34e67fc700 1 -- 172.21.15.84:0/3478662601 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f34e004c1d0 con 0x7f34f00f56c0 2024-01-18T07:35:17.331 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:17.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 msgr2=0x7f34dc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:17.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 0x7f34dc05e760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f34ec009d60 tx=0x7f34ec009b30).stop 2024-01-18T07:35:17.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 msgr2=0x7f34f0109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:17.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f34e00093f0 tx=0x7f34e0006ec0).stop 2024-01-18T07:35:17.332 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 shutdown_connections 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f34f00f56c0 0x7f34f0109df0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f34dc05c2a0 0x7f34dc05e760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 --2- 172.21.15.84:0/3478662601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f34f00f6960 0x7f34f010c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 >> 172.21.15.84:0/3478662601 conn(0x7f34f00f0e20 msgr2=0x7f34f00fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 shutdown_connections 2024-01-18T07:35:17.333 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:17.332+0000 7f34f81c9700 1 -- 172.21.15.84:0/3478662601 wait complete. 2024-01-18T07:35:17.351 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:17.014916+0000 mgr.smithi084.sihbqg (mgr.14162) 531 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:17.014916+0000 mgr.smithi084.sihbqg (mgr.14162) 531 : cluster [DBG] pgmap v497: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:17.742 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:17.331249+0000 mon.smithi084 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.84:0/3478662601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:18.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:17.331249+0000 mon.smithi084 (mon.0) 513 : audit [DBG] from='client.? 172.21.15.84:0/3478662601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:19.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:19.015335+0000 mgr.smithi084.sihbqg (mgr.14162) 532 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:19.015335+0000 mgr.smithi084.sihbqg (mgr.14162) 532 : cluster [DBG] pgmap v498: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:20.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:20.977 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:21.015720+0000 mgr.smithi084.sihbqg (mgr.14162) 533 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:21.015720+0000 mgr.smithi084.sihbqg (mgr.14162) 533 : cluster [DBG] pgmap v499: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:21.986 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 -- 172.21.15.84:0/3793703136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 msgr2=0x7f53140f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 --2- 172.21.15.84:0/3793703136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53140f7ec0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f5308004660 tx=0x7f5308009b30).stop 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 -- 172.21.15.84:0/3793703136 shutdown_connections 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 --2- 172.21.15.84:0/3793703136 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53140f7ec0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 --2- 172.21.15.84:0/3793703136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 0x7f53140f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 -- 172.21.15.84:0/3793703136 >> 172.21.15.84:0/3793703136 conn(0x7f53140f1f20 msgr2=0x7f53140f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 -- 172.21.15.84:0/3793703136 shutdown_connections 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.984+0000 7f5318eb3700 1 -- 172.21.15.84:0/3793703136 wait complete. 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 Processor -- start 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 -- start start 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 0x7f53141070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:21.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314107b50 con 0x7f53140f7a60 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5318eb3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314107c90 con 0x7f53140f67c0 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5311d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5311d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55924/0 (socket says 172.21.15.84:55924) 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f5311d9b700 1 -- 172.21.15.84:0/2430950365 learned_addr learned my addr 172.21.15.84:0/2430950365 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.985+0000 7f531259c700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 0x7f53141070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5311d9b700 1 -- 172.21.15.84:0/2430950365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 msgr2=0x7f53141070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5311d9b700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 0x7f53141070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5311d9b700 1 -- 172.21.15.84:0/2430950365 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5308005040 con 0x7f53140f7a60 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5311d9b700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f5308009f50 tx=0x7f5308009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5308018070 con 0x7f53140f7a60 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53141085d0 con 0x7f53140f7a60 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5308013410 con 0x7f53140f7a60 2024-01-18T07:35:21.988 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5314108b30 con 0x7f53140f7a60 2024-01-18T07:35:21.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.986+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f530801c3c0 con 0x7f53140f7a60 2024-01-18T07:35:21.989 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.987+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5314045ad0 con 0x7f53140f7a60 2024-01-18T07:35:21.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.987+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f530801d420 con 0x7f53140f7a60 2024-01-18T07:35:21.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.987+0000 7f5302ffd700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 0x7f52fc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:21.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.987+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f5308077800 con 0x7f53140f7a60 2024-01-18T07:35:21.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.988+0000 7f531259c700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 0x7f52fc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:21.990 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.988+0000 7f531259c700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 0x7f52fc05e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f53140f78c0 tx=0x7f530400b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:21.991 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:21.991+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f530804a800 con 0x7f53140f7a60 2024-01-18T07:35:22.200 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.200+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5314027a00 con 0x7f53140f7a60 2024-01-18T07:35:22.200 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.200+0000 7f5302ffd700 1 -- 172.21.15.84:0/2430950365 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f530804de20 con 0x7f53140f7a60 2024-01-18T07:35:22.201 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 msgr2=0x7f52fc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 0x7f52fc05e760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f53140f78c0 tx=0x7f530400b040).stop 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 msgr2=0x7f53141075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f5308009f50 tx=0x7f5308009b30).stop 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 shutdown_connections 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f52fc05c2a0 0x7f52fc05e760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53140f67c0 0x7f53141070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 --2- 172.21.15.84:0/2430950365 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f53140f7a60 0x7f53141075a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:22.202 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 >> 172.21.15.84:0/2430950365 conn(0x7f53140f1f20 msgr2=0x7f53140fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:22.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 shutdown_connections 2024-01-18T07:35:22.203 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:22.202+0000 7f5318eb3700 1 -- 172.21.15.84:0/2430950365 wait complete. 2024-01-18T07:35:22.491 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:22.201102+0000 mon.smithi084 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.84:0/2430950365' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:22.575 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:22.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:22.201102+0000 mon.smithi084 (mon.0) 514 : audit [DBG] from='client.? 172.21.15.84:0/2430950365' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:23.633 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:23.016049+0000 mgr.smithi084.sihbqg (mgr.14162) 534 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:23.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:23.016049+0000 mgr.smithi084.sihbqg (mgr.14162) 534 : cluster [DBG] pgmap v500: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:25.016490+0000 mgr.smithi084.sihbqg (mgr.14162) 535 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:25.016490+0000 mgr.smithi084.sihbqg (mgr.14162) 535 : cluster [DBG] pgmap v501: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:25.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:25.809 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:26.771 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 -- 172.21.15.84:0/4246403057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f6060 msgr2=0x7f62ec0f64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 --2- 172.21.15.84:0/4246403057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f6060 0x7f62ec0f64e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f62e8004770 tx=0x7f62e800ac40).stop 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 -- 172.21.15.84:0/4246403057 shutdown_connections 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 --2- 172.21.15.84:0/4246403057 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f6060 0x7f62ec0f64e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 --2- 172.21.15.84:0/4246403057 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f4e60 0x7f62ec0f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 -- 172.21.15.84:0/4246403057 >> 172.21.15.84:0/4246403057 conn(0x7f62ec0f05e0 msgr2=0x7f62ec0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 -- 172.21.15.84:0/4246403057 shutdown_connections 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.769+0000 7f62f4261700 1 -- 172.21.15.84:0/4246403057 wait complete. 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 Processor -- start 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 -- start start 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 0x7f62ec1075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec107be0 con 0x7f62ec0f4e60 2024-01-18T07:35:26.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f4261700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62ec108490 con 0x7f62ec0f6060 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55812/0 (socket says 172.21.15.84:55812) 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 -- 172.21.15.84:0/1905053123 learned_addr learned my addr 172.21.15.84:0/1905053123 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f17fc700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 0x7f62ec1075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 -- 172.21.15.84:0/1905053123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 msgr2=0x7f62ec1075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 0x7f62ec1075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 -- 172.21.15.84:0/1905053123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e8005040 con 0x7f62ec0f4e60 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f17fc700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 0x7f62ec1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.770+0000 7f62f1ffd700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f62e000b410 tx=0x7f62e0006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.771+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62e00125f0 con 0x7f62ec0f4e60 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.771+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62e0012c00 con 0x7f62ec0f4e60 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.771+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec108770 con 0x7f62ec0f4e60 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.771+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62e0002aa0 con 0x7f62ec0f4e60 2024-01-18T07:35:26.773 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.771+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62ec108d30 con 0x7f62ec0f4e60 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.772+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62ec059100 con 0x7f62ec0f4e60 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.772+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62e000f030 con 0x7f62ec0f4e60 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.772+0000 7f62de7fc700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 0x7f62d805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.772+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f62e0076060 con 0x7f62ec0f4e60 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.772+0000 7f62f17fc700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 0x7f62d805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:26.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.773+0000 7f62f17fc700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 0x7f62d805e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f62e80043d0 tx=0x7f62e800ac40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:26.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.776+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62e0049000 con 0x7f62ec0f4e60 2024-01-18T07:35:26.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.981+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f62ec045ad0 con 0x7f62ec0f4e60 2024-01-18T07:35:26.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.982+0000 7f62de7fc700 1 -- 172.21.15.84:0/1905053123 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f62e004c620 con 0x7f62ec0f4e60 2024-01-18T07:35:26.982 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:26.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 msgr2=0x7f62d805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:26.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 0x7f62d805e760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f62e80043d0 tx=0x7f62e800ac40).stop 2024-01-18T07:35:26.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 msgr2=0x7f62ec1070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:26.984 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f62e000b410 tx=0x7f62e0006ef0).stop 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 shutdown_connections 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62ec0f4e60 0x7f62ec1070b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62d805c2a0 0x7f62d805e760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 --2- 172.21.15.84:0/1905053123 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62ec0f6060 0x7f62ec1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.983+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 >> 172.21.15.84:0/1905053123 conn(0x7f62ec0f05e0 msgr2=0x7f62ec0f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.984+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 shutdown_connections 2024-01-18T07:35:26.987 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:26.984+0000 7f62f4261700 1 -- 172.21.15.84:0/1905053123 wait complete. 2024-01-18T07:35:27.323 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:26.982995+0000 mon.smithi084 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.84:0/1905053123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:27.323 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:27.016859+0000 mgr.smithi084.sihbqg (mgr.14162) 536 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:27.374 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:26.982995+0000 mon.smithi084 (mon.0) 515 : audit [DBG] from='client.? 172.21.15.84:0/1905053123' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:27.016859+0000 mgr.smithi084.sihbqg (mgr.14162) 536 : cluster [DBG] pgmap v502: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:29.017236+0000 mgr.smithi084.sihbqg (mgr.14162) 537 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:29.017236+0000 mgr.smithi084.sihbqg (mgr.14162) 537 : cluster [DBG] pgmap v503: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:30.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:30.608 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:31.017565+0000 mgr.smithi084.sihbqg (mgr.14162) 538 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:31.017565+0000 mgr.smithi084.sihbqg (mgr.14162) 538 : cluster [DBG] pgmap v504: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.605+0000 7f365eff9700 1 -- 172.21.15.84:0/3784226457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 msgr2=0x7f36580f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.605+0000 7f365eff9700 1 --2- 172.21.15.84:0/3784226457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f36580f6a30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3654004660 tx=0x7f3654009b30).stop 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- 172.21.15.84:0/3784226457 shutdown_connections 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 --2- 172.21.15.84:0/3784226457 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 0x7f36580f7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 --2- 172.21.15.84:0/3784226457 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f36580f6a30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- 172.21.15.84:0/3784226457 >> 172.21.15.84:0/3784226457 conn(0x7f36580f1d70 msgr2=0x7f36580f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- 172.21.15.84:0/3784226457 shutdown_connections 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- 172.21.15.84:0/3784226457 wait complete. 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 Processor -- start 2024-01-18T07:35:31.608 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- start start 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 0x7f3658107420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3658107a60 con 0x7f36580f6610 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365eff9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3658108310 con 0x7f36580f78b0 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365dff7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365dff7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:55840/0 (socket says 172.21.15.84:55840) 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.606+0000 7f365dff7700 1 -- 172.21.15.84:0/2600106414 learned_addr learned my addr 172.21.15.84:0/2600106414 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365d7f6700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 0x7f3658107420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365dff7700 1 -- 172.21.15.84:0/2600106414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 msgr2=0x7f3658107420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365dff7700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 0x7f3658107420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365dff7700 1 -- 172.21.15.84:0/2600106414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3654005040 con 0x7f36580f6610 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365dff7700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3654009f50 tx=0x7f3654009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3654018070 con 0x7f36580f6610 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3658108590 con 0x7f36580f6610 2024-01-18T07:35:31.609 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3654013410 con 0x7f36580f6610 2024-01-18T07:35:31.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f365401c620 con 0x7f36580f6610 2024-01-18T07:35:31.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.607+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3658108af0 con 0x7f36580f6610 2024-01-18T07:35:31.610 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.608+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3658059100 con 0x7f36580f6610 2024-01-18T07:35:31.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.608+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f365401d420 con 0x7f36580f6610 2024-01-18T07:35:31.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.608+0000 7f364e7fc700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 0x7f364405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:31.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.608+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3654077a40 con 0x7f36580f6610 2024-01-18T07:35:31.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.609+0000 7f365d7f6700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 0x7f364405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:31.611 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.609+0000 7f365d7f6700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 0x7f364405e770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f364800b790 tx=0x7f3648008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:31.612 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.612+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f365404aa40 con 0x7f36580f6610 2024-01-18T07:35:31.820 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.820+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3658027a00 con 0x7f36580f6610 2024-01-18T07:35:31.821 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.820+0000 7f364e7fc700 1 -- 172.21.15.84:0/2600106414 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f365404e060 con 0x7f36580f6610 2024-01-18T07:35:31.821 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:31.822 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 msgr2=0x7f364405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:31.822 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 0x7f364405e770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f364800b790 tx=0x7f3648008040).stop 2024-01-18T07:35:31.822 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 msgr2=0x7f3658106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:31.822 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f3654009f50 tx=0x7f3654009b30).stop 2024-01-18T07:35:31.822 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 shutdown_connections 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36580f6610 0x7f3658106f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f364405c2b0 0x7f364405e770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 --2- 172.21.15.84:0/2600106414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36580f78b0 0x7f3658107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 >> 172.21.15.84:0/2600106414 conn(0x7f36580f1d70 msgr2=0x7f36580fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 shutdown_connections 2024-01-18T07:35:31.823 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:31.822+0000 7f365eff9700 1 -- 172.21.15.84:0/2600106414 wait complete. 2024-01-18T07:35:32.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:31.821412+0000 mon.smithi084 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.84:0/2600106414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:32.239 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:32.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:31.821412+0000 mon.smithi084 (mon.0) 516 : audit [DBG] from='client.? 172.21.15.84:0/2600106414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:33.018208+0000 mgr.smithi084.sihbqg (mgr.14162) 539 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:33.018208+0000 mgr.smithi084.sihbqg (mgr.14162) 539 : cluster [DBG] pgmap v505: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:35.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:35.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:35.018557+0000 mgr.smithi084.sihbqg (mgr.14162) 540 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:35.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:35.053513+0000 mon.smithi084 (mon.0) 517 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:35:35.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:35.064199+0000 mon.smithi084 (mon.0) 518 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:35:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:35.018557+0000 mgr.smithi084.sihbqg (mgr.14162) 540 : cluster [DBG] pgmap v506: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:35.053513+0000 mon.smithi084 (mon.0) 517 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:35:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:35.064199+0000 mon.smithi084 (mon.0) 518 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:35:35.476 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:36.487 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- 172.21.15.84:0/3797971172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 msgr2=0x7f89800f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3797971172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f89800f6dc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f897c004660 tx=0x7f897c009b30).stop 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- 172.21.15.84:0/3797971172 shutdown_connections 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3797971172 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f89800f6dc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3797971172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f89800f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- 172.21.15.84:0/3797971172 >> 172.21.15.84:0/3797971172 conn(0x7f89800f0e20 msgr2=0x7f89800f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- 172.21.15.84:0/3797971172 shutdown_connections 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- 172.21.15.84:0/3797971172 wait complete. 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 Processor -- start 2024-01-18T07:35:36.488 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.487+0000 7f8987a9a700 1 -- start start 2024-01-18T07:35:36.490 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f8980109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:36.490 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:36.490 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980060920 con 0x7f89800f6960 2024-01-18T07:35:36.490 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980060a60 con 0x7f89800f56c0 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54712/0 (socket says 172.21.15.84:54712) 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 -- 172.21.15.84:0/3806260423 learned_addr learned my addr 172.21.15.84:0/3806260423 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985836700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f8980109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 -- 172.21.15.84:0/3806260423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 msgr2=0x7f8980109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f8980109e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 -- 172.21.15.84:0/3806260423 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897c005040 con 0x7f89800f6960 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985836700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f8980109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8985035700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f897c009f50 tx=0x7f897c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c018070 con 0x7f89800f6960 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f897c013410 con 0x7f89800f6960 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898010c850 con 0x7f89800f6960 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f897c01c3c0 con 0x7f89800f6960 2024-01-18T07:35:36.491 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.488+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898010cd80 con 0x7f89800f6960 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.489+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980059100 con 0x7f89800f6960 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.490+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f897c01d420 con 0x7f89800f6960 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.490+0000 7f89767fc700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 0x7f896c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.490+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f897c0777c0 con 0x7f89800f6960 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.490+0000 7f8985836700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 0x7f896c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:36.492 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.490+0000 7f8985836700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 0x7f896c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f89800605a0 tx=0x7f8970009c00).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:36.494 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.494+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f897c04a8f0 con 0x7f89800f6960 2024-01-18T07:35:36.700 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.700+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8980045ad0 con 0x7f89800f6960 2024-01-18T07:35:36.701 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.700+0000 7f89767fc700 1 -- 172.21.15.84:0/3806260423 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f897c04df10 con 0x7f89800f6960 2024-01-18T07:35:36.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:36.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 msgr2=0x7f896c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:36.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 0x7f896c05e760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f89800605a0 tx=0x7f8970009c00).stop 2024-01-18T07:35:36.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 msgr2=0x7f898010c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:36.702 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f897c009f50 tx=0x7f897c009b30).stop 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 shutdown_connections 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f896c05c2a0 0x7f896c05e760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89800f56c0 0x7f8980109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 --2- 172.21.15.84:0/3806260423 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f89800f6960 0x7f898010c300 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 >> 172.21.15.84:0/3806260423 conn(0x7f89800f0e20 msgr2=0x7f89800fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 shutdown_connections 2024-01-18T07:35:36.703 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:36.702+0000 7f8987a9a700 1 -- 172.21.15.84:0/3806260423 wait complete. 2024-01-18T07:35:37.074 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:36.701374+0000 mon.smithi084 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.84:0/3806260423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:37.019095+0000 mgr.smithi084.sihbqg (mgr.14162) 541 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:36.701374+0000 mon.smithi084 (mon.0) 519 : audit [DBG] from='client.? 172.21.15.84:0/3806260423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:37.019095+0000 mgr.smithi084.sihbqg (mgr.14162) 541 : cluster [DBG] pgmap v507: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:39.019532+0000 mgr.smithi084.sihbqg (mgr.14162) 542 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:39.019532+0000 mgr.smithi084.sihbqg (mgr.14162) 542 : cluster [DBG] pgmap v508: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:40.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:40.307 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:41.249 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.248+0000 7f64e529b700 1 -- 172.21.15.84:0/1390080850 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 msgr2=0x7f64e00f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.248+0000 7f64e529b700 1 --2- 172.21.15.84:0/1390080850 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e00f6750 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f64d0004660 tx=0x7f64d0009b30).stop 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 -- 172.21.15.84:0/1390080850 shutdown_connections 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 --2- 172.21.15.84:0/1390080850 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e00f6750 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 --2- 172.21.15.84:0/1390080850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 0x7f64e00f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 -- 172.21.15.84:0/1390080850 >> 172.21.15.84:0/1390080850 conn(0x7f64e00f1d70 msgr2=0x7f64e00f41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 -- 172.21.15.84:0/1390080850 shutdown_connections 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 -- 172.21.15.84:0/1390080850 wait complete. 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 Processor -- start 2024-01-18T07:35:41.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.249+0000 7f64e529b700 1 -- start start 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 0x7f64e0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e006a380 con 0x7f64e00f62f0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e006a4c0 con 0x7f64e00f7ce0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54738/0 (socket says 172.21.15.84:54738) 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 -- 172.21.15.84:0/2116006767 learned_addr learned my addr 172.21.15.84:0/2116006767 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64df7fe700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 0x7f64e0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 -- 172.21.15.84:0/2116006767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 msgr2=0x7f64e0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 0x7f64e0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 -- 172.21.15.84:0/2116006767 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0005040 con 0x7f64e00f62f0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dffff700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f64d400c530 tx=0x7f64d4007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d400f650 con 0x7f64e00f62f0 2024-01-18T07:35:41.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e006a7a0 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64d400fc60 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64d4012d20 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.250+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e010efd0 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.251+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0045ad0 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.252+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f64d4019030 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.252+0000 7f64dcff9700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 0x7f64c805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.252+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f64d4075830 con 0x7f64e00f62f0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.252+0000 7f64df7fe700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 0x7f64c805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:41.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.252+0000 7f64df7fe700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 0x7f64c805e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f64d0013580 tx=0x7f64d0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:41.256 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.256+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64d4048880 con 0x7f64e00f62f0 2024-01-18T07:35:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:41.019875+0000 mgr.smithi084.sihbqg (mgr.14162) 543 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:41.019875+0000 mgr.smithi084.sihbqg (mgr.14162) 543 : cluster [DBG] pgmap v509: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:41.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.463+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f64e006a930 con 0x7f64e00f62f0 2024-01-18T07:35:41.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.463+0000 7f64dcff9700 1 -- 172.21.15.84:0/2116006767 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f64e006a930 con 0x7f64e00f62f0 2024-01-18T07:35:41.464 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:41.465 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 msgr2=0x7f64c805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 0x7f64c805e770 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f64d0013580 tx=0x7f64d0009b30).stop 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 msgr2=0x7f64e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f64d400c530 tx=0x7f64d4007e10).stop 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 shutdown_connections 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f64e00f62f0 0x7f64e006b9a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f64c805c2b0 0x7f64c805e770 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 --2- 172.21.15.84:0/2116006767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64e00f7ce0 0x7f64e0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 >> 172.21.15.84:0/2116006767 conn(0x7f64e00f1d70 msgr2=0x7f64e00fb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 shutdown_connections 2024-01-18T07:35:41.466 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:41.465+0000 7f64e529b700 1 -- 172.21.15.84:0/2116006767 wait complete. 2024-01-18T07:35:41.854 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:42.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:41.464601+0000 mon.smithi084 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.84:0/2116006767' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:42.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:41.464601+0000 mon.smithi084 (mon.0) 520 : audit [DBG] from='client.? 172.21.15.84:0/2116006767' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:43.020200+0000 mgr.smithi084.sihbqg (mgr.14162) 544 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:43.020200+0000 mgr.smithi084.sihbqg (mgr.14162) 544 : cluster [DBG] pgmap v510: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:44.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:45.089 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:45.256 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:45.020488+0000 mgr.smithi084.sihbqg (mgr.14162) 545 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:45.020488+0000 mgr.smithi084.sihbqg (mgr.14162) 545 : cluster [DBG] pgmap v511: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:46.126 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 -- 172.21.15.84:0/1414775073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 msgr2=0x7f07980f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1414775073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07980f6dc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f0794004660 tx=0x7f0794009b30).stop 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 -- 172.21.15.84:0/1414775073 shutdown_connections 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1414775073 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07980f6dc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1414775073 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 0x7f07980f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 -- 172.21.15.84:0/1414775073 >> 172.21.15.84:0/1414775073 conn(0x7f07980f0e20 msgr2=0x7f07980f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 -- 172.21.15.84:0/1414775073 shutdown_connections 2024-01-18T07:35:46.127 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.125+0000 7f079f7fd700 1 -- 172.21.15.84:0/1414775073 wait complete. 2024-01-18T07:35:46.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 Processor -- start 2024-01-18T07:35:46.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 -- start start 2024-01-18T07:35:46.129 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 0x7f0798105fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0798106a80 con 0x7f07980f6960 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079f7fd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0798106bc0 con 0x7f07980f56c0 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54760/0 (socket says 172.21.15.84:54760) 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 -- 172.21.15.84:0/1461323903 learned_addr learned my addr 172.21.15.84:0/1461323903 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079d599700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 0x7f0798105fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 -- 172.21.15.84:0/1461323903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 msgr2=0x7f0798105fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 0x7f0798105fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.126+0000 7f079cd98700 1 -- 172.21.15.84:0/1461323903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0794005040 con 0x7f07980f6960 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f079cd98700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f0794009830 tx=0x7f0794009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0794018070 con 0x7f07980f6960 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0794013410 con 0x7f07980f6960 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0798107500 con 0x7f07980f6960 2024-01-18T07:35:46.130 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f079401c620 con 0x7f07980f6960 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.127+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0798107a60 con 0x7f07980f6960 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.128+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0798045ad0 con 0x7f07980f6960 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.128+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f079401c8e0 con 0x7f07980f6960 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.129+0000 7f0789ffb700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 0x7f078405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.129+0000 7f079d599700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 0x7f078405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.129+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f0794077a60 con 0x7f07980f6960 2024-01-18T07:35:46.131 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.129+0000 7f079d599700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 0x7f078405e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f07980f67c0 tx=0x7f078c009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:46.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.133+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f079404aa00 con 0x7f07980f6960 2024-01-18T07:35:46.340 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.338+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0798027a00 con 0x7f07980f6960 2024-01-18T07:35:46.340 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.339+0000 7f0789ffb700 1 -- 172.21.15.84:0/1461323903 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f079404e020 con 0x7f07980f6960 2024-01-18T07:35:46.340 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:46.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 msgr2=0x7f078405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 0x7f078405e760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f07980f67c0 tx=0x7f078c009040).stop 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 msgr2=0x7f07981064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f0794009830 tx=0x7f0794009b30).stop 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 shutdown_connections 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f078405c2a0 0x7f078405e760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f07980f56c0 0x7f0798105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 --2- 172.21.15.84:0/1461323903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f07980f6960 0x7f07981064d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 >> 172.21.15.84:0/1461323903 conn(0x7f07980f0e20 msgr2=0x7f07980fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 shutdown_connections 2024-01-18T07:35:46.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:46.340+0000 7f079f7fd700 1 -- 172.21.15.84:0/1461323903 wait complete. 2024-01-18T07:35:46.737 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:46.340733+0000 mon.smithi084 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.84:0/1461323903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:47.020798+0000 mgr.smithi084.sihbqg (mgr.14162) 546 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:46.340733+0000 mon.smithi084 (mon.0) 521 : audit [DBG] from='client.? 172.21.15.84:0/1461323903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:47.020798+0000 mgr.smithi084.sihbqg (mgr.14162) 546 : cluster [DBG] pgmap v512: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:49.021138+0000 mgr.smithi084.sihbqg (mgr.14162) 547 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:49.021138+0000 mgr.smithi084.sihbqg (mgr.14162) 547 : cluster [DBG] pgmap v513: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:49.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:49.971 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 -- 172.21.15.84:0/759081933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980607f0 msgr2=0x7f29980f9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 --2- 172.21.15.84:0/759081933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980607f0 0x7f29980f9280 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2988004660 tx=0x7f2988009b30).stop 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 -- 172.21.15.84:0/759081933 shutdown_connections 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 --2- 172.21.15.84:0/759081933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29980f9800 0x7f29980fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 --2- 172.21.15.84:0/759081933 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980607f0 0x7f29980f9280 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 -- 172.21.15.84:0/759081933 >> 172.21.15.84:0/759081933 conn(0x7f29980f0ca0 msgr2=0x7f29980f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 -- 172.21.15.84:0/759081933 shutdown_connections 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.021+0000 7f299759e700 1 -- 172.21.15.84:0/759081933 wait complete. 2024-01-18T07:35:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 Processor -- start 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 -- start start 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29980607f0 0x7f2998106f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29981079a0 con 0x7f29980f9800 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2998107ae0 con 0x7f29980607f0 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54464/0 (socket says 172.21.15.84:54464) 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 -- 172.21.15.84:0/3723789135 learned_addr learned my addr 172.21.15.84:0/3723789135 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 -- 172.21.15.84:0/3723789135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29980607f0 msgr2=0x7f2998106f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29980607f0 0x7f2998106f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 -- 172.21.15.84:0/3723789135 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988005040 con 0x7f29980f9800 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2995d9b700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f298c00c510 tx=0x7f298c007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f298c00f650 con 0x7f29980f9800 2024-01-18T07:35:51.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998108480 con 0x7f29980f9800 2024-01-18T07:35:51.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f298c00fc60 con 0x7f29980f9800 2024-01-18T07:35:51.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f298c012c80 con 0x7f29980f9800 2024-01-18T07:35:51.026 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.022+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2998108a40 con 0x7f29980f9800 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.023+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2998045ad0 con 0x7f29980f9800 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.024+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f298c010070 con 0x7f29980f9800 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.024+0000 7f2986ffd700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 0x7f298005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.024+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f298c0758b0 con 0x7f29980f9800 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.024+0000 7f299659c700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 0x7f298005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:51.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.024+0000 7f299659c700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 0x7f298005e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2988009f50 tx=0x7f2988009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:51.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.028+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f298c0452b0 con 0x7f29980f9800 2024-01-18T07:35:51.237 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.236+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2998027a00 con 0x7f29980f9800 2024-01-18T07:35:51.237 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.236+0000 7f2986ffd700 1 -- 172.21.15.84:0/3723789135 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f298c0488b0 con 0x7f29980f9800 2024-01-18T07:35:51.237 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:51.239 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 msgr2=0x7f298005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:51.239 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 0x7f298005e770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2988009f50 tx=0x7f2988009b30).stop 2024-01-18T07:35:51.239 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 msgr2=0x7f29981073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:51.239 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f298c00c510 tx=0x7f298c007df0).stop 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 shutdown_connections 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f298005c2b0 0x7f298005e770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29980607f0 0x7f2998106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 --2- 172.21.15.84:0/3723789135 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f29980f9800 0x7f29981073f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 >> 172.21.15.84:0/3723789135 conn(0x7f29980f0ca0 msgr2=0x7f29980f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 shutdown_connections 2024-01-18T07:35:51.240 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:51.238+0000 7f299759e700 1 -- 172.21.15.84:0/3723789135 wait complete. 2024-01-18T07:35:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:51.021473+0000 mgr.smithi084.sihbqg (mgr.14162) 548 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:51.021473+0000 mgr.smithi084.sihbqg (mgr.14162) 548 : cluster [DBG] pgmap v514: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:51.660 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:52.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:51.238051+0000 mon.smithi084 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.84:0/3723789135' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:52.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:51.238051+0000 mon.smithi084 (mon.0) 522 : audit [DBG] from='client.? 172.21.15.84:0/3723789135' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:53.021928+0000 mgr.smithi084.sihbqg (mgr.14162) 549 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:53.021928+0000 mgr.smithi084.sihbqg (mgr.14162) 549 : cluster [DBG] pgmap v515: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:54.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:54.895 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:35:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:55.022175+0000 mgr.smithi084.sihbqg (mgr.14162) 550 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:55.022175+0000 mgr.smithi084.sihbqg (mgr.14162) 550 : cluster [DBG] pgmap v516: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 -- 172.21.15.84:0/500102724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f5510 msgr2=0x7fabac0f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 --2- 172.21.15.84:0/500102724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f5510 0x7fabac0f5930 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fab9c004660 tx=0x7fab9c009b30).stop 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 -- 172.21.15.84:0/500102724 shutdown_connections 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 --2- 172.21.15.84:0/500102724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f67b0 0x7fabac0f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 --2- 172.21.15.84:0/500102724 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f5510 0x7fabac0f5930 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:55.925 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 -- 172.21.15.84:0/500102724 >> 172.21.15.84:0/500102724 conn(0x7fabac0f0c70 msgr2=0x7fabac0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:55.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 -- 172.21.15.84:0/500102724 shutdown_connections 2024-01-18T07:35:55.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.923+0000 7fabb2edd700 1 -- 172.21.15.84:0/500102724 wait complete. 2024-01-18T07:35:55.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 Processor -- start 2024-01-18T07:35:55.926 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 -- start start 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 0x7fabac0f9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0f8900 con 0x7fabac0f67b0 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb2edd700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0f8a40 con 0x7fabac0f5510 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb16da700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb16da700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54470/0 (socket says 172.21.15.84:54470) 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.924+0000 7fabb16da700 1 -- 172.21.15.84:0/3456152718 learned_addr learned my addr 172.21.15.84:0/3456152718 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb1edb700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 0x7fabac0f9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb16da700 1 -- 172.21.15.84:0/3456152718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 msgr2=0x7fabac0f9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb16da700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 0x7fabac0f9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb16da700 1 -- 172.21.15.84:0/3456152718 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c005040 con 0x7fabac0f67b0 2024-01-18T07:35:55.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb1edb700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 0x7fabac0f9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb16da700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7faba800b410 tx=0x7faba8006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba80125f0 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faba8012c00 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac0f8d20 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faba8002aa0 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.925+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac10ef20 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.926+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac045ad0 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.926+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7faba800f030 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.926+0000 7faba27fc700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 0x7fab9805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.927+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7faba8076060 con 0x7fabac0f67b0 2024-01-18T07:35:55.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.927+0000 7fabb1edb700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 0x7fab9805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:35:55.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.927+0000 7fabb1edb700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 0x7fab9805e770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fab9c009d60 tx=0x7fab9c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:35:55.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:55.930+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faba8049000 con 0x7fabac0f67b0 2024-01-18T07:35:56.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.136+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fabac0fdd20 con 0x7fabac0f67b0 2024-01-18T07:35:56.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.136+0000 7faba27fc700 1 -- 172.21.15.84:0/3456152718 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7faba804c620 con 0x7fabac0f67b0 2024-01-18T07:35:56.138 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 msgr2=0x7fab9805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 0x7fab9805e770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fab9c009d60 tx=0x7fab9c009b30).stop 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 msgr2=0x7fabac0f8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7faba800b410 tx=0x7faba8006ef0).stop 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 shutdown_connections 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab9805c2b0 0x7fab9805e770 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fabac0f5510 0x7fabac0f9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 --2- 172.21.15.84:0/3456152718 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fabac0f67b0 0x7fabac0f8310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 >> 172.21.15.84:0/3456152718 conn(0x7fabac0f0c70 msgr2=0x7fabac05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 shutdown_connections 2024-01-18T07:35:56.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:35:56.138+0000 7fabb2edd700 1 -- 172.21.15.84:0/3456152718 wait complete. 2024-01-18T07:35:56.561 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:35:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:35:56.138394+0000 mon.smithi084 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.84:0/3456152718' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:57.022597+0000 mgr.smithi084.sihbqg (mgr.14162) 551 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:35:56.138394+0000 mon.smithi084 (mon.0) 523 : audit [DBG] from='client.? 172.21.15.84:0/3456152718' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:35:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:57.022597+0000 mgr.smithi084.sihbqg (mgr.14162) 551 : cluster [DBG] pgmap v517: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:35:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:35:59.023090+0000 mgr.smithi084.sihbqg (mgr.14162) 552 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:35:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:35:59.023090+0000 mgr.smithi084.sihbqg (mgr.14162) 552 : cluster [DBG] pgmap v518: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:35:59.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:35:59.796 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.725+0000 7f8c7260d700 1 -- 172.21.15.84:0/1233197602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 msgr2=0x7f8c6c0f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:00.727 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.725+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1233197602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c0f5930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f8c68004660 tx=0x7f8c68009b30).stop 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- 172.21.15.84:0/1233197602 shutdown_connections 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1233197602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c6c0f67b0 0x7f8c6c0f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1233197602 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c0f5930 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- 172.21.15.84:0/1233197602 >> 172.21.15.84:0/1233197602 conn(0x7f8c6c0f0c70 msgr2=0x7f8c6c0f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- 172.21.15.84:0/1233197602 shutdown_connections 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- 172.21.15.84:0/1233197602 wait complete. 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 Processor -- start 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- start start 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c6c0f67b0 0x7f8c6c10c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0609d0 con 0x7f8c6c0f5510 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7260d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c060b10 con 0x7f8c6c0f67b0 2024-01-18T07:36:00.728 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7160b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7160b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40368/0 (socket says 172.21.15.84:40368) 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.726+0000 7f8c7160b700 1 -- 172.21.15.84:0/1789496268 learned_addr learned my addr 172.21.15.84:0/1789496268 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7160b700 1 -- 172.21.15.84:0/1789496268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c6c0f67b0 msgr2=0x7f8c6c10c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7160b700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c6c0f67b0 0x7f8c6c10c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7160b700 1 -- 172.21.15.84:0/1789496268 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68005040 con 0x7f8c6c0f5510 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7160b700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f8c68009f50 tx=0x7f8c68009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c68018070 con 0x7f8c6c0f5510 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c68013410 con 0x7f8c6c0f5510 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c10c700 con 0x7f8c6c0f5510 2024-01-18T07:36:00.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c6801c620 con 0x7f8c6c0f5510 2024-01-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.727+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c10cc30 con 0x7f8c6c0f5510 2024-01-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.728+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c059100 con 0x7f8c6c0f5510 2024-01-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.728+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8c6801d420 con 0x7f8c6c0f5510 2024-01-18T07:36:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.729+0000 7f8c61ffb700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 0x7f8c5805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.729+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8c68077a00 con 0x7f8c6c0f5510 2024-01-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.729+0000 7f8c70e0a700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 0x7f8c5805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:00.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.729+0000 7f8c70e0a700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 0x7f8c5805e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8c5c009610 tx=0x7f8c5c006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:00.733 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.733+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c6804aa00 con 0x7f8c6c0f5510 2024-01-18T07:36:00.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.940+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8c6c045ad0 con 0x7f8c6c0f5510 2024-01-18T07:36:00.940 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.940+0000 7f8c61ffb700 1 -- 172.21.15.84:0/1789496268 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8c6804e020 con 0x7f8c6c0f5510 2024-01-18T07:36:00.940 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 msgr2=0x7f8c5805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 0x7f8c5805e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8c5c009610 tx=0x7f8c5c006ab0).stop 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 msgr2=0x7f8c6c109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f8c68009f50 tx=0x7f8c68009b30).stop 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 shutdown_connections 2024-01-18T07:36:00.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8c6c0f5510 0x7f8c6c109c00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:00.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8c5805c2b0 0x7f8c5805e770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:00.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 --2- 172.21.15.84:0/1789496268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8c6c0f67b0 0x7f8c6c10c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:00.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 >> 172.21.15.84:0/1789496268 conn(0x7f8c6c0f0c70 msgr2=0x7f8c6c0fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:00.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 shutdown_connections 2024-01-18T07:36:00.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:00.942+0000 7f8c7260d700 1 -- 172.21.15.84:0/1789496268 wait complete. 2024-01-18T07:36:01.257 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:00.941052+0000 mon.smithi084 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.84:0/1789496268' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:01.325 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:00.941052+0000 mon.smithi084 (mon.0) 524 : audit [DBG] from='client.? 172.21.15.84:0/1789496268' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:02.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:01.023366+0000 mgr.smithi084.sihbqg (mgr.14162) 553 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:01.023366+0000 mgr.smithi084.sihbqg (mgr.14162) 553 : cluster [DBG] pgmap v519: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:03.023876+0000 mgr.smithi084.sihbqg (mgr.14162) 554 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:03.023876+0000 mgr.smithi084.sihbqg (mgr.14162) 554 : cluster [DBG] pgmap v520: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:04.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:04.561 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:05.332 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:05.024240+0000 mgr.smithi084.sihbqg (mgr.14162) 555 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:05.024240+0000 mgr.smithi084.sihbqg (mgr.14162) 555 : cluster [DBG] pgmap v521: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:05.687 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 -- 172.21.15.84:0/3638853516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f56e0 msgr2=0x7fb1280f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 --2- 172.21.15.84:0/3638853516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f56e0 0x7fb1280f5b00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fb118004660 tx=0x7fb118009b30).stop 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 -- 172.21.15.84:0/3638853516 shutdown_connections 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 --2- 172.21.15.84:0/3638853516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f6980 0x7fb1280f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 --2- 172.21.15.84:0/3638853516 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f56e0 0x7fb1280f5b00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 -- 172.21.15.84:0/3638853516 >> 172.21.15.84:0/3638853516 conn(0x7fb1280f0e20 msgr2=0x7fb1280f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 -- 172.21.15.84:0/3638853516 shutdown_connections 2024-01-18T07:36:05.688 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.687+0000 7fb12cda7700 1 -- 172.21.15.84:0/3638853516 wait complete. 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 Processor -- start 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 -- start start 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f56e0 0x7fb128106000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb128106b30 con 0x7fb1280f6980 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12cda7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1281073e0 con 0x7fb1280f56e0 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:40380/0 (socket says 172.21.15.84:40380) 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 -- 172.21.15.84:0/2337476345 learned_addr learned my addr 172.21.15.84:0/2337476345 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb12659c700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f56e0 0x7fb128106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 -- 172.21.15.84:0/2337476345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f56e0 msgr2=0x7fb128106000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:05.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f56e0 0x7fb128106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 -- 172.21.15.84:0/2337476345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118005040 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb125d9b700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fb11000b410 tx=0x7fb110006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.688+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1100125f0 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.689+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb110012c00 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.689+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1281076c0 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.689+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb110002aa0 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.689+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb128107c80 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.689+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb128059100 con 0x7fb1280f6980 2024-01-18T07:36:05.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.690+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb11000f030 con 0x7fb1280f6980 2024-01-18T07:36:05.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.690+0000 7fb11effd700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 0x7fb11405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:05.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.690+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb110076060 con 0x7fb1280f6980 2024-01-18T07:36:05.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.690+0000 7fb12659c700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 0x7fb11405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:05.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.690+0000 7fb12659c700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 0x7fb11405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb118004e30 tx=0x7fb118009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:05.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.694+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb110049000 con 0x7fb1280f6980 2024-01-18T07:36:05.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.901+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb1280fdef0 con 0x7fb1280f6980 2024-01-18T07:36:05.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.901+0000 7fb11effd700 1 -- 172.21.15.84:0/2337476345 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb11004c620 con 0x7fb1280f6980 2024-01-18T07:36:05.902 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 msgr2=0x7fb11405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 0x7fb11405e760 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb118004e30 tx=0x7fb118009b30).stop 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 msgr2=0x7fb1281064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fb11000b410 tx=0x7fb110006ef0).stop 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 shutdown_connections 2024-01-18T07:36:05.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb11405c2a0 0x7fb11405e760 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:05.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1280f56e0 0x7fb128106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:05.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 --2- 172.21.15.84:0/2337476345 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb1280f6980 0x7fb1281064f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:05.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 >> 172.21.15.84:0/2337476345 conn(0x7fb1280f0e20 msgr2=0x7fb1280fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:05.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 shutdown_connections 2024-01-18T07:36:05.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:05.903+0000 7fb12cda7700 1 -- 172.21.15.84:0/2337476345 wait complete. 2024-01-18T07:36:06.228 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:05.902172+0000 mon.smithi084 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.84:0/2337476345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:06.281 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:06.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:05.902172+0000 mon.smithi084 (mon.0) 525 : audit [DBG] from='client.? 172.21.15.84:0/2337476345' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:07.024581+0000 mgr.smithi084.sihbqg (mgr.14162) 556 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:07.024581+0000 mgr.smithi084.sihbqg (mgr.14162) 556 : cluster [DBG] pgmap v522: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:09.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:09.024968+0000 mgr.smithi084.sihbqg (mgr.14162) 557 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:09.024968+0000 mgr.smithi084.sihbqg (mgr.14162) 557 : cluster [DBG] pgmap v523: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:09.516 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:10.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3479116480 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 msgr2=0x7f1e2c0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:10.566 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3479116480 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c0f7ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f1e20004660 tx=0x7f1e20009b30).stop 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3479116480 shutdown_connections 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3479116480 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c0f7ed0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3479116480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 0x7f1e2c0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3479116480 >> 172.21.15.84:0/3479116480 conn(0x7f1e2c0f1f50 msgr2=0x7f1e2c0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3479116480 shutdown_connections 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.562+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3479116480 wait complete. 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 Processor -- start 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 -- start start 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 0x7f1e2c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:10.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c06a410 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e30b2d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c06a550 con 0x7f1e2c0f67d0 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e29d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e29d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:39216/0 (socket says 172.21.15.84:39216) 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e29d9b700 1 -- 172.21.15.84:0/3344993349 learned_addr learned my addr 172.21.15.84:0/3344993349 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.563+0000 7f1e2a59c700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 0x7f1e2c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e29d9b700 1 -- 172.21.15.84:0/3344993349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 msgr2=0x7f1e2c06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e29d9b700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 0x7f1e2c06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:10.568 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e29d9b700 1 -- 172.21.15.84:0/3344993349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e20005040 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e29d9b700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f1e20009830 tx=0x7f1e20009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e20018070 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c06a7d0 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e20013410 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2001c400 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.564+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c10f0f0 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.565+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c045ad0 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.565+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f1e2001d420 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.565+0000 7f1e1affd700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 0x7f1e1405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.566+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f1e20077a00 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.566+0000 7f1e2a59c700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 0x7f1e1405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:10.569 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.566+0000 7f1e2a59c700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 0x7f1e1405e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1e2c06b730 tx=0x7f1e1c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:10.570 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.570+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e2004aa00 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.776+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1e2c027a00 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.777+0000 7f1e1affd700 1 -- 172.21.15.84:0/3344993349 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f1e2004e020 con 0x7f1e2c0f7a70 2024-01-18T07:36:10.777 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 msgr2=0x7f1e1405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 0x7f1e1405e760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1e2c06b730 tx=0x7f1e1c008040).stop 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 msgr2=0x7f1e2c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f1e20009830 tx=0x7f1e20009b30).stop 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 shutdown_connections 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f1e1405c2a0 0x7f1e1405e760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e2c0f67d0 0x7f1e2c06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 --2- 172.21.15.84:0/3344993349 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f1e2c0f7a70 0x7f1e2c069e20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.778+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 >> 172.21.15.84:0/3344993349 conn(0x7f1e2c0f1f50 msgr2=0x7f1e2c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.779+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 shutdown_connections 2024-01-18T07:36:10.779 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:10.779+0000 7f1e30b2d700 1 -- 172.21.15.84:0/3344993349 wait complete. 2024-01-18T07:36:11.080 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:10.777842+0000 mon.smithi084 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.84:0/3344993349' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:11.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:10.777842+0000 mon.smithi084 (mon.0) 526 : audit [DBG] from='client.? 172.21.15.84:0/3344993349' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:11.189 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:12.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:11.025258+0000 mgr.smithi084.sihbqg (mgr.14162) 558 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:12.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:11.025258+0000 mgr.smithi084.sihbqg (mgr.14162) 558 : cluster [DBG] pgmap v524: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:13.025616+0000 mgr.smithi084.sihbqg (mgr.14162) 559 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:13.025616+0000 mgr.smithi084.sihbqg (mgr.14162) 559 : cluster [DBG] pgmap v525: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:14.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:14.424 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:15.251 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:15.025890+0000 mgr.smithi084.sihbqg (mgr.14162) 560 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:15.025890+0000 mgr.smithi084.sihbqg (mgr.14162) 560 : cluster [DBG] pgmap v526: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 -- 172.21.15.84:0/1286476665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 msgr2=0x7fca400f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 --2- 172.21.15.84:0/1286476665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca400f6be0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fca3c004660 tx=0x7fca3c009b30).stop 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 -- 172.21.15.84:0/1286476665 shutdown_connections 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 --2- 172.21.15.84:0/1286476665 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 0x7fca400f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 --2- 172.21.15.84:0/1286476665 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca400f6be0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 -- 172.21.15.84:0/1286476665 >> 172.21.15.84:0/1286476665 conn(0x7fca400f1f20 msgr2=0x7fca400f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 -- 172.21.15.84:0/1286476665 shutdown_connections 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.437+0000 7fca472c3700 1 -- 172.21.15.84:0/1286476665 wait complete. 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 Processor -- start 2024-01-18T07:36:15.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 -- start start 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 0x7fca40107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40107b40 con 0x7fca400f67c0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca472c3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40107c80 con 0x7fca400f7a60 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca4505f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca4505f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:39244/0 (socket says 172.21.15.84:39244) 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.438+0000 7fca4505f700 1 -- 172.21.15.84:0/2953310397 learned_addr learned my addr 172.21.15.84:0/2953310397 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca4485e700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 0x7fca40107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca4505f700 1 -- 172.21.15.84:0/2953310397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 msgr2=0x7fca40107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca4505f700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 0x7fca40107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca4505f700 1 -- 172.21.15.84:0/2953310397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c005040 con 0x7fca400f67c0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca4505f700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fca3c009f50 tx=0x7fca3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c018070 con 0x7fca400f67c0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca3c013410 con 0x7fca400f67c0 2024-01-18T07:36:15.441 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca401085c0 con 0x7fca400f67c0 2024-01-18T07:36:15.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3c01c3c0 con 0x7fca400f67c0 2024-01-18T07:36:15.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.439+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca40108b20 con 0x7fca400f67c0 2024-01-18T07:36:15.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.440+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca40045ad0 con 0x7fca400f67c0 2024-01-18T07:36:15.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.440+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fca3c01d420 con 0x7fca400f67c0 2024-01-18T07:36:15.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.441+0000 7fca31ffb700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 0x7fca2c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:15.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.441+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fca3c0778f0 con 0x7fca400f67c0 2024-01-18T07:36:15.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.441+0000 7fca4485e700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 0x7fca2c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:15.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.441+0000 7fca4485e700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 0x7fca2c05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fca340041d0 tx=0x7fca34009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:15.445 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.445+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca3c04a8f0 con 0x7fca400f67c0 2024-01-18T07:36:15.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.652+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca40027a00 con 0x7fca400f67c0 2024-01-18T07:36:15.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.652+0000 7fca31ffb700 1 -- 172.21.15.84:0/2953310397 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fca3c04df10 con 0x7fca400f67c0 2024-01-18T07:36:15.653 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.653+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 msgr2=0x7fca2c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.653+0000 7fca472c3700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 0x7fca2c05e760 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fca340041d0 tx=0x7fca34009040).stop 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.653+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 msgr2=0x7fca401070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.653+0000 7fca472c3700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fca3c009f50 tx=0x7fca3c009b30).stop 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 shutdown_connections 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fca400f67c0 0x7fca401070a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fca2c05c2a0 0x7fca2c05e760 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 --2- 172.21.15.84:0/2953310397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca400f7a60 0x7fca40107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:15.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 >> 172.21.15.84:0/2953310397 conn(0x7fca400f1f20 msgr2=0x7fca400fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:15.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 shutdown_connections 2024-01-18T07:36:15.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:15.654+0000 7fca472c3700 1 -- 172.21.15.84:0/2953310397 wait complete. 2024-01-18T07:36:16.061 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:15.653083+0000 mon.smithi084 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.84:0/2953310397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:16.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:15.653083+0000 mon.smithi084 (mon.0) 527 : audit [DBG] from='client.? 172.21.15.84:0/2953310397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:17.026252+0000 mgr.smithi084.sihbqg (mgr.14162) 561 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:17.026252+0000 mgr.smithi084.sihbqg (mgr.14162) 561 : cluster [DBG] pgmap v527: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:36:17.280435+0000 mgr.smithi084.sihbqg (mgr.14162) 562 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:36:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:34:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:35:02Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:35:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:36:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:36:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:34:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:35:02Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr time="2024-01-18T07:35:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:36:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('uid/gid not found') 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: uid/gid not found 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:36:17.281161+0000 mgr.smithi084.sihbqg (mgr.14162) 563 : cephadm [INF] Reconfiguring node-exporter.smithi106 (unknown last config time)... 2024-01-18T07:36:18.386 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:36:17.281294+0000 mgr.smithi084.sihbqg (mgr.14162) 564 : cephadm [INF] Reconfiguring daemon node-exporter.smithi106 on smithi106 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:36:17.280435+0000 mgr.smithi084.sihbqg (mgr.14162) 562 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:34:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:35:02Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:35:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:36:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:36:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi106 ... 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi106 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:34:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:35:02Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr time="2024-01-18T07:35:39Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-18T07:36:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('uid/gid not found') 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: uid/gid not found 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:36:17.281161+0000 mgr.smithi084.sihbqg (mgr.14162) 563 : cephadm [INF] Reconfiguring node-exporter.smithi106 (unknown last config time)... 2024-01-18T07:36:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:36:17.281294+0000 mgr.smithi084.sihbqg (mgr.14162) 564 : cephadm [INF] Reconfiguring daemon node-exporter.smithi106 on smithi106 2024-01-18T07:36:19.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:19.299 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:19.381 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:19.026734+0000 mgr.smithi084.sihbqg (mgr.14162) 565 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:19.026734+0000 mgr.smithi084.sihbqg (mgr.14162) 565 : cluster [DBG] pgmap v528: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:20.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 -- 172.21.15.84:0/413892825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f56c0 msgr2=0x7effec0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:20.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 --2- 172.21.15.84:0/413892825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f56c0 0x7effec0f5ae0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7effe0004660 tx=0x7effe0009b30).stop 2024-01-18T07:36:20.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 -- 172.21.15.84:0/413892825 shutdown_connections 2024-01-18T07:36:20.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 --2- 172.21.15.84:0/413892825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f6960 0x7effec0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:20.221 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 --2- 172.21.15.84:0/413892825 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f56c0 0x7effec0f5ae0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 -- 172.21.15.84:0/413892825 >> 172.21.15.84:0/413892825 conn(0x7effec0f0e20 msgr2=0x7effec0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 -- 172.21.15.84:0/413892825 shutdown_connections 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.219+0000 7efff1fba700 1 -- 172.21.15.84:0/413892825 wait complete. 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 Processor -- start 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 -- start start 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f56c0 0x7effec0fa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec0f8ab0 con 0x7effec0f6960 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7efff1fba700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec0f8bf0 con 0x7effec0f56c0 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:39560/0 (socket says 172.21.15.84:39560) 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 -- 172.21.15.84:0/3505329055 learned_addr learned my addr 172.21.15.84:0/3505329055 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeb7fe700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f56c0 0x7effec0fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:20.222 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 -- 172.21.15.84:0/3505329055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f56c0 msgr2=0x7effec0fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f56c0 0x7effec0fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.220+0000 7effeaffd700 1 -- 172.21.15.84:0/3505329055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe0005040 con 0x7effec0f6960 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7effeaffd700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7effdc00b410 tx=0x7effdc006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effdc0125f0 con 0x7effec0f6960 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effdc00be30 con 0x7effec0f6960 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec0f8ed0 con 0x7effec0f6960 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effdc002a20 con 0x7effec0f6960 2024-01-18T07:36:20.223 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.221+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec10f120 con 0x7effec0f6960 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.222+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec045ad0 con 0x7effec0f6960 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.222+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7effdc00f030 con 0x7effec0f6960 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.222+0000 7effd3fff700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 0x7effd405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.222+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7effdc075ea0 con 0x7effec0f6960 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.222+0000 7effeb7fe700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 0x7effd405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:20.224 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.223+0000 7effeb7fe700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 0x7effd405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7effec0f9dd0 tx=0x7effe0009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:20.226 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.226+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effdc048f20 con 0x7effec0f6960 2024-01-18T07:36:20.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.430+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7effec10f8b0 con 0x7effec0f6960 2024-01-18T07:36:20.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.431+0000 7effd3fff700 1 -- 172.21.15.84:0/3505329055 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7effdc04c540 con 0x7effec0f6960 2024-01-18T07:36:20.431 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 msgr2=0x7effd405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 0x7effd405e760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7effec0f9dd0 tx=0x7effe0009b30).stop 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 msgr2=0x7effec0f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7effdc00b410 tx=0x7effdc006ef0).stop 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 shutdown_connections 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7effd405c2a0 0x7effd405e760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7effec0f56c0 0x7effec0fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 --2- 172.21.15.84:0/3505329055 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7effec0f6960 0x7effec0f84c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.432+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 >> 172.21.15.84:0/3505329055 conn(0x7effec0f0e20 msgr2=0x7effec05fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.433+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 shutdown_connections 2024-01-18T07:36:20.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:20.433+0000 7efff1fba700 1 -- 172.21.15.84:0/3505329055 wait complete. 2024-01-18T07:36:20.778 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:20.431933+0000 mon.smithi084 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.84:0/3505329055' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:20.822 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:20.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:20.431933+0000 mon.smithi084 (mon.0) 528 : audit [DBG] from='client.? 172.21.15.84:0/3505329055' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:21.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:21.027229+0000 mgr.smithi084.sihbqg (mgr.14162) 566 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:21.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:21.027229+0000 mgr.smithi084.sihbqg (mgr.14162) 566 : cluster [DBG] pgmap v529: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:23.027643+0000 mgr.smithi084.sihbqg (mgr.14162) 567 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:23.027643+0000 mgr.smithi084.sihbqg (mgr.14162) 567 : cluster [DBG] pgmap v530: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:23.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:24.055 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:25.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2905095844 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f78b0 msgr2=0x7f32ac0f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2905095844 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f78b0 0x7f32ac0f7d10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f3298004660 tx=0x7f3298009b30).stop 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2905095844 shutdown_connections 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2905095844 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f78b0 0x7f32ac0f7d10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2905095844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f6610 0x7f32ac0f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2905095844 >> 172.21.15.84:0/2905095844 conn(0x7f32ac0f1d70 msgr2=0x7f32ac0f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2905095844 shutdown_connections 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2905095844 wait complete. 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 Processor -- start 2024-01-18T07:36:25.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.071+0000 7f32b1d7e700 1 -- start start 2024-01-18T07:36:25.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:25.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f78b0 0x7f32ac069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:25.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac06a380 con 0x7f32ac0f6610 2024-01-18T07:36:25.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac06a4c0 con 0x7f32ac0f78b0 2024-01-18T07:36:25.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:39578/0 (socket says 172.21.15.84:39578) 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 -- 172.21.15.84:0/2460604211 learned_addr learned my addr 172.21.15.84:0/2460604211 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32abfff700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f78b0 0x7f32ac069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 -- 172.21.15.84:0/2460604211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f78b0 msgr2=0x7f32ac069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f78b0 0x7f32ac069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 -- 172.21.15.84:0/2460604211 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298005040 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b0d7c700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f32a000c790 tx=0x7f32a0007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a0013630 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac06a7a0 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32a0012860 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32a00027e0 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.072+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac10ef00 con 0x7f32ac0f6610 2024-01-18T07:36:25.075 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.073+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac045ad0 con 0x7f32ac0f6610 2024-01-18T07:36:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.074+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f32a0010070 con 0x7f32ac0f6610 2024-01-18T07:36:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.074+0000 7f32a97fa700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 0x7f329c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.074+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f32a0075c40 con 0x7f32ac0f6610 2024-01-18T07:36:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.074+0000 7f32abfff700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 0x7f329c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:25.076 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.074+0000 7f32abfff700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 0x7f329c05e770 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3298004630 tx=0x7f3298009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:25.078 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.078+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32a0048be0 con 0x7f32ac0f6610 2024-01-18T07:36:25.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.287+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32ac0fee70 con 0x7f32ac0f6610 2024-01-18T07:36:25.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.287+0000 7f32a97fa700 1 -- 172.21.15.84:0/2460604211 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f32a004c200 con 0x7f32ac0f6610 2024-01-18T07:36:25.288 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.288+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 msgr2=0x7f329c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.288+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 0x7f329c05e770 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3298004630 tx=0x7f3298009b30).stop 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.288+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 msgr2=0x7f32ac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.288+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f32a000c790 tx=0x7f32a0007d90).stop 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 shutdown_connections 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f32ac0f6610 0x7f32ac06b9a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f329c05c2b0 0x7f329c05e770 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 --2- 172.21.15.84:0/2460604211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32ac0f78b0 0x7f32ac069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 >> 172.21.15.84:0/2460604211 conn(0x7f32ac0f1d70 msgr2=0x7f32ac0fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:25.289 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 shutdown_connections 2024-01-18T07:36:25.290 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:25.289+0000 7f32b1d7e700 1 -- 172.21.15.84:0/2460604211 wait complete. 2024-01-18T07:36:25.307 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:25.027953+0000 mgr.smithi084.sihbqg (mgr.14162) 568 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:25.027953+0000 mgr.smithi084.sihbqg (mgr.14162) 568 : cluster [DBG] pgmap v531: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:25.678 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:26.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:25.288053+0000 mon.smithi084 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.84:0/2460604211' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:26.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:25.288053+0000 mon.smithi084 (mon.0) 529 : audit [DBG] from='client.? 172.21.15.84:0/2460604211' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:27.028292+0000 mgr.smithi084.sihbqg (mgr.14162) 569 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:27.028292+0000 mgr.smithi084.sihbqg (mgr.14162) 569 : cluster [DBG] pgmap v532: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:28.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:28.916 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:29.028611+0000 mgr.smithi084.sihbqg (mgr.14162) 570 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:29.028611+0000 mgr.smithi084.sihbqg (mgr.14162) 570 : cluster [DBG] pgmap v533: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:29.928 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 -- 172.21.15.84:0/2056360398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f67c0 msgr2=0x7f62d00f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 --2- 172.21.15.84:0/2056360398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f67c0 0x7f62d00f6be0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f62c00045a0 tx=0x7f62c0009b30).stop 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 -- 172.21.15.84:0/2056360398 shutdown_connections 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 --2- 172.21.15.84:0/2056360398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f7a60 0x7f62d00f7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 --2- 172.21.15.84:0/2056360398 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f67c0 0x7f62d00f6be0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 -- 172.21.15.84:0/2056360398 >> 172.21.15.84:0/2056360398 conn(0x7f62d00f1f20 msgr2=0x7f62d00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 -- 172.21.15.84:0/2056360398 shutdown_connections 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.926+0000 7f62d831b700 1 -- 172.21.15.84:0/2056360398 wait complete. 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 Processor -- start 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 -- start start 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 0x7f62d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d006a380 con 0x7f62d00f7a60 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d006a4c0 con 0x7f62d00f67c0 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:29.929 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56286/0 (socket says 172.21.15.84:56286) 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 -- 172.21.15.84:0/3845412016 learned_addr learned my addr 172.21.15.84:0/3845412016 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d60b7700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 0x7f62d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 -- 172.21.15.84:0/3845412016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 msgr2=0x7f62d006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 0x7f62d006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 -- 172.21.15.84:0/3845412016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc005040 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d60b7700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 0x7f62d006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d58b6700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f62cc004960 tx=0x7f62cc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc007e50 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62cc00f040 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c0005040 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.927+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc0149b0 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.928+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d006ab50 con 0x7f62d00f7a60 2024-01-18T07:36:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.928+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d0059100 con 0x7f62d00f7a60 2024-01-18T07:36:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.929+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f62cc014be0 con 0x7f62d00f7a60 2024-01-18T07:36:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.929+0000 7f62c6ffd700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 0x7f62bc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.929+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f62cc075d40 con 0x7f62d00f7a60 2024-01-18T07:36:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.929+0000 7f62d60b7700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 0x7f62bc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.929+0000 7f62d60b7700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 0x7f62bc05e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f62c00133e0 tx=0x7f62c000b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:29.935 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:29.933+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62cc0472b0 con 0x7f62d00f7a60 2024-01-18T07:36:30.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.142+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f62d0045ad0 con 0x7f62d00f7a60 2024-01-18T07:36:30.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.142+0000 7f62c6ffd700 1 -- 172.21.15.84:0/3845412016 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f62cc048d40 con 0x7f62d00f7a60 2024-01-18T07:36:30.143 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:30.144 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 msgr2=0x7f62bc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 0x7f62bc05e760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f62c00133e0 tx=0x7f62c000b040).stop 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 msgr2=0x7f62d0069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f62cc004960 tx=0x7f62cc009b30).stop 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 shutdown_connections 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f62bc05c2a0 0x7f62bc05e760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62d00f67c0 0x7f62d006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 --2- 172.21.15.84:0/3845412016 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f62d00f7a60 0x7f62d0069e20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 >> 172.21.15.84:0/3845412016 conn(0x7f62d00f1f20 msgr2=0x7f62d00fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 shutdown_connections 2024-01-18T07:36:30.145 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:30.144+0000 7f62d831b700 1 -- 172.21.15.84:0/3845412016 wait complete. 2024-01-18T07:36:30.517 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:30.143463+0000 mon.smithi084 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.84:0/3845412016' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:30.560 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:30.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:30.143463+0000 mon.smithi084 (mon.0) 530 : audit [DBG] from='client.? 172.21.15.84:0/3845412016' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:31.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:31.028919+0000 mgr.smithi084.sihbqg (mgr.14162) 571 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:31.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:31.028919+0000 mgr.smithi084.sihbqg (mgr.14162) 571 : cluster [DBG] pgmap v534: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:33.029285+0000 mgr.smithi084.sihbqg (mgr.14162) 572 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:33.029285+0000 mgr.smithi084.sihbqg (mgr.14162) 572 : cluster [DBG] pgmap v535: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:33.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:33.793 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:34.826 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 -- 172.21.15.84:0/7862957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 msgr2=0x7f85980f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 --2- 172.21.15.84:0/7862957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f85980f5930 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f858c004660 tx=0x7f858c009b30).stop 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 -- 172.21.15.84:0/7862957 shutdown_connections 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 --2- 172.21.15.84:0/7862957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 0x7f85980f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 --2- 172.21.15.84:0/7862957 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f85980f5930 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 -- 172.21.15.84:0/7862957 >> 172.21.15.84:0/7862957 conn(0x7f85980f0c70 msgr2=0x7f85980f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 -- 172.21.15.84:0/7862957 shutdown_connections 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.824+0000 7f859d863700 1 -- 172.21.15.84:0/7862957 wait complete. 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 Processor -- start 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 -- start start 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 0x7f859810c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:34.827 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85980609d0 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859d863700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8598060b10 con 0x7f85980f67b0 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859c861700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859c861700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56302/0 (socket says 172.21.15.84:56302) 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f859c861700 1 -- 172.21.15.84:0/3010311170 learned_addr learned my addr 172.21.15.84:0/3010311170 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.825+0000 7f8597fff700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 0x7f859810c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859c861700 1 -- 172.21.15.84:0/3010311170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 msgr2=0x7f859810c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859c861700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 0x7f859810c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859c861700 1 -- 172.21.15.84:0/3010311170 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c005040 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859c861700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f858c009f50 tx=0x7f858c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c018070 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859810c700 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f858c013410 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858c01c620 con 0x7f85980f5510 2024-01-18T07:36:34.828 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.826+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859810cc30 con 0x7f85980f5510 2024-01-18T07:36:34.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.827+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598059100 con 0x7f85980f5510 2024-01-18T07:36:34.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.827+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f858c01d420 con 0x7f85980f5510 2024-01-18T07:36:34.829 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.827+0000 7f85957fa700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 0x7f858805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:34.830 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.827+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f858c077a00 con 0x7f85980f5510 2024-01-18T07:36:34.830 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.827+0000 7f8597fff700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 0x7f858805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:34.830 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.828+0000 7f8597fff700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 0x7f858805e770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f858400b790 tx=0x7f8584008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:34.831 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:34.831+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f858c04aa00 con 0x7f85980f5510 2024-01-18T07:36:35.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.037+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8598045ad0 con 0x7f85980f5510 2024-01-18T07:36:35.037 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.037+0000 7f85957fa700 1 -- 172.21.15.84:0/3010311170 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f858c04e020 con 0x7f85980f5510 2024-01-18T07:36:35.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 msgr2=0x7f858805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 0x7f858805e770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f858400b790 tx=0x7f8584008040).stop 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 msgr2=0x7f8598109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f858c009f50 tx=0x7f858c009b30).stop 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 shutdown_connections 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f85980f5510 0x7f8598109c00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:35.039 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f858805c2b0 0x7f858805e770 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:35.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 --2- 172.21.15.84:0/3010311170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85980f67b0 0x7f859810c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:35.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 >> 172.21.15.84:0/3010311170 conn(0x7f85980f0c70 msgr2=0x7f85980fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:35.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 shutdown_connections 2024-01-18T07:36:35.040 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:35.039+0000 7f859d863700 1 -- 172.21.15.84:0/3010311170 wait complete. 2024-01-18T07:36:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:35.029581+0000 mgr.smithi084.sihbqg (mgr.14162) 573 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:35.038271+0000 mon.smithi084 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.84:0/3010311170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:35.055224+0000 mon.smithi084 (mon.0) 532 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:36:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:35.065532+0000 mon.smithi084 (mon.0) 533 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:36:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:35.029581+0000 mgr.smithi084.sihbqg (mgr.14162) 573 : cluster [DBG] pgmap v536: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:35.038271+0000 mon.smithi084 (mon.0) 531 : audit [DBG] from='client.? 172.21.15.84:0/3010311170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:35.055224+0000 mon.smithi084 (mon.0) 532 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:36:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:35.065532+0000 mon.smithi084 (mon.0) 533 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:36:35.443 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:37.030061+0000 mgr.smithi084.sihbqg (mgr.14162) 574 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:37.030061+0000 mgr.smithi084.sihbqg (mgr.14162) 574 : cluster [DBG] pgmap v537: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:38.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:38.677 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:39.030464+0000 mgr.smithi084.sihbqg (mgr.14162) 575 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:39.030464+0000 mgr.smithi084.sihbqg (mgr.14162) 575 : cluster [DBG] pgmap v538: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 -- 172.21.15.84:0/3521495393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 msgr2=0x7f70840f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 --2- 172.21.15.84:0/3521495393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f70840f5ae0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7078004660 tx=0x7f7078009b30).stop 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 -- 172.21.15.84:0/3521495393 shutdown_connections 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 --2- 172.21.15.84:0/3521495393 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70840f6960 0x7f70840f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 --2- 172.21.15.84:0/3521495393 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f70840f5ae0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 -- 172.21.15.84:0/3521495393 >> 172.21.15.84:0/3521495393 conn(0x7f70840f0e20 msgr2=0x7f70840f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 -- 172.21.15.84:0/3521495393 shutdown_connections 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.597+0000 7f708b305700 1 -- 172.21.15.84:0/3521495393 wait complete. 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 Processor -- start 2024-01-18T07:36:39.599 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 -- start start 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70840f6960 0x7f708410c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840609b0 con 0x7f70840f56c0 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7084060af0 con 0x7f70840f6960 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58832/0 (socket says 172.21.15.84:58832) 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 -- 172.21.15.84:0/2080538631 learned_addr learned my addr 172.21.15.84:0/2080538631 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 -- 172.21.15.84:0/2080538631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70840f6960 msgr2=0x7f708410c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70840f6960 0x7f708410c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 -- 172.21.15.84:0/2080538631 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078005040 con 0x7f70840f56c0 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f70890a1700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f7078009f50 tx=0x7f7078009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7078018070 con 0x7f70840f56c0 2024-01-18T07:36:39.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7078013410 con 0x7f70840f56c0 2024-01-18T07:36:39.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f707801c620 con 0x7f70840f56c0 2024-01-18T07:36:39.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.598+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708410c8a0 con 0x7f70840f56c0 2024-01-18T07:36:39.601 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.599+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708410ce00 con 0x7f70840f56c0 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.599+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7084045ad0 con 0x7f70840f56c0 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.600+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f707801d420 con 0x7f70840f56c0 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.600+0000 7f7075ffb700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 0x7f707005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.600+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f70780779a0 con 0x7f70840f56c0 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.600+0000 7f70888a0700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 0x7f707005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:39.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.601+0000 7f70888a0700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 0x7f707005e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7080009610 tx=0x7f7080006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:39.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.604+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f707804a9c0 con 0x7f70840f56c0 2024-01-18T07:36:39.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.812+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7084027a00 con 0x7f70840f56c0 2024-01-18T07:36:39.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.812+0000 7f7075ffb700 1 -- 172.21.15.84:0/2080538631 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f707804dfe0 con 0x7f70840f56c0 2024-01-18T07:36:39.813 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 msgr2=0x7f707005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 0x7f707005e760 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7080009610 tx=0x7f7080006ab0).stop 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 msgr2=0x7f7084109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f7078009f50 tx=0x7f7078009b30).stop 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 shutdown_connections 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f70840f56c0 0x7f7084109df0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f707005c2a0 0x7f707005e760 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 --2- 172.21.15.84:0/2080538631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f70840f6960 0x7f708410c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 >> 172.21.15.84:0/2080538631 conn(0x7f70840f0e20 msgr2=0x7f70840fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 shutdown_connections 2024-01-18T07:36:39.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:39.814+0000 7f708b305700 1 -- 172.21.15.84:0/2080538631 wait complete. 2024-01-18T07:36:40.191 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:40.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:39.813657+0000 mon.smithi084 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.84:0/2080538631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:40.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:39.813657+0000 mon.smithi084 (mon.0) 534 : audit [DBG] from='client.? 172.21.15.84:0/2080538631' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:41.030956+0000 mgr.smithi084.sihbqg (mgr.14162) 576 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:41.030956+0000 mgr.smithi084.sihbqg (mgr.14162) 576 : cluster [DBG] pgmap v539: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:43.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:43.031398+0000 mgr.smithi084.sihbqg (mgr.14162) 577 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:43.031398+0000 mgr.smithi084.sihbqg (mgr.14162) 577 : cluster [DBG] pgmap v540: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:43.426 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:44.477 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.474+0000 7fa193b33700 1 -- 172.21.15.84:0/3376105082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f6960 msgr2=0x7fa18c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.474+0000 7fa193b33700 1 --2- 172.21.15.84:0/3376105082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f6960 0x7fa18c0f6dc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fa1880045a0 tx=0x7fa188009b30).stop 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.474+0000 7fa193b33700 1 -- 172.21.15.84:0/3376105082 shutdown_connections 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 --2- 172.21.15.84:0/3376105082 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f6960 0x7fa18c0f6dc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 --2- 172.21.15.84:0/3376105082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f56c0 0x7fa18c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- 172.21.15.84:0/3376105082 >> 172.21.15.84:0/3376105082 conn(0x7fa18c0f0e20 msgr2=0x7fa18c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- 172.21.15.84:0/3376105082 shutdown_connections 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- 172.21.15.84:0/3376105082 wait complete. 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 Processor -- start 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- start start 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f6960 0x7fa18c106410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c1069c0 con 0x7fa18c0f56c0 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa193b33700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c106b00 con 0x7fa18c0f6960 2024-01-18T07:36:44.478 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa1918cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa1918cf700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58852/0 (socket says 172.21.15.84:58852) 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa1918cf700 1 -- 172.21.15.84:0/3731657489 learned_addr learned my addr 172.21.15.84:0/3731657489 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.475+0000 7fa1910ce700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f6960 0x7fa18c106410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1918cf700 1 -- 172.21.15.84:0/3731657489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f6960 msgr2=0x7fa18c106410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1918cf700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f6960 0x7fa18c106410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1918cf700 1 -- 172.21.15.84:0/3731657489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c005040 con 0x7fa18c0f56c0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1918cf700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa17c004960 tx=0x7fa17c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c00e030 con 0x7fa18c0f56c0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa17c00f040 con 0x7fa18c0f56c0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa188005040 con 0x7fa18c0f56c0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa17c0148a0 con 0x7fa18c0f56c0 2024-01-18T07:36:44.479 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.476+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c107850 con 0x7fa18c0f56c0 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.477+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c045ad0 con 0x7fa18c0f56c0 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.477+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fa17c010500 con 0x7fa18c0f56c0 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.478+0000 7fa1827fc700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 0x7fa17805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.478+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fa17c076190 con 0x7fa18c0f56c0 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.478+0000 7fa1910ce700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 0x7fa17805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:44.480 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.478+0000 7fa1910ce700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 0x7fa17805e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa188000c00 tx=0x7fa188016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:44.483 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.481+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa17c049190 con 0x7fa18c0f56c0 2024-01-18T07:36:44.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.690+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa18c0fde50 con 0x7fa18c0f56c0 2024-01-18T07:36:44.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.691+0000 7fa1827fc700 1 -- 172.21.15.84:0/3731657489 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fa17c04c7b0 con 0x7fa18c0f56c0 2024-01-18T07:36:44.692 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 msgr2=0x7fa17805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 0x7fa17805e760 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fa188000c00 tx=0x7fa188016040).stop 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 msgr2=0x7fa18c105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fa17c004960 tx=0x7fa17c009b30).stop 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 shutdown_connections 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fa18c0f56c0 0x7fa18c105f20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fa17805c2a0 0x7fa17805e760 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.694 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 --2- 172.21.15.84:0/3731657489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa18c0f6960 0x7fa18c106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:44.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.692+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 >> 172.21.15.84:0/3731657489 conn(0x7fa18c0f0e20 msgr2=0x7fa18c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:44.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.693+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 shutdown_connections 2024-01-18T07:36:44.695 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:44.693+0000 7fa193b33700 1 -- 172.21.15.84:0/3731657489 wait complete. 2024-01-18T07:36:45.041 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:44 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:44.692758+0000 mon.smithi084 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.84:0/3731657489' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:45.090 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:45.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:44 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:44.692758+0000 mon.smithi084 (mon.0) 535 : audit [DBG] from='client.? 172.21.15.84:0/3731657489' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:46.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:45.031858+0000 mgr.smithi084.sihbqg (mgr.14162) 578 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:46.164 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:45.031858+0000 mgr.smithi084.sihbqg (mgr.14162) 578 : cluster [DBG] pgmap v541: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:47.032250+0000 mgr.smithi084.sihbqg (mgr.14162) 579 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:47.032250+0000 mgr.smithi084.sihbqg (mgr.14162) 579 : cluster [DBG] pgmap v542: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:48.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:48.326 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:49.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:49.032650+0000 mgr.smithi084.sihbqg (mgr.14162) 580 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1196744475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 msgr2=0x7f2fd40f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:49.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1196744475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd40f6bf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f2fc8004660 tx=0x7f2fc8009b30).stop 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1196744475 shutdown_connections 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1196744475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 0x7f2fd40f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1196744475 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd40f6bf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1196744475 >> 172.21.15.84:0/1196744475 conn(0x7f2fd40f1f50 msgr2=0x7f2fd40f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1196744475 shutdown_connections 2024-01-18T07:36:49.366 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1196744475 wait complete. 2024-01-18T07:36:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.364+0000 7f2fdb76a700 1 Processor -- start 2024-01-18T07:36:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 -- start start 2024-01-18T07:36:49.368 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 0x7f2fd41075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4107b50 con 0x7f2fd40f67d0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4107c90 con 0x7f2fd40f7a70 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59324/0 (socket says 172.21.15.84:59324) 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 -- 172.21.15.84:0/1087387881 learned_addr learned my addr 172.21.15.84:0/1087387881 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd8d05700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 0x7f2fd41075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 -- 172.21.15.84:0/1087387881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 msgr2=0x7f2fd41075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 0x7f2fd41075a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 -- 172.21.15.84:0/1087387881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc8005040 con 0x7f2fd40f67d0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fd9506700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f2fc8009f50 tx=0x7f2fc8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc8018070 con 0x7f2fd40f67d0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fc8013410 con 0x7f2fd40f67d0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd41085d0 con 0x7f2fd40f67d0 2024-01-18T07:36:49.369 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.365+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2fc801c620 con 0x7f2fd40f67d0 2024-01-18T07:36:49.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.366+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd4108b30 con 0x7f2fd40f67d0 2024-01-18T07:36:49.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.366+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4045ad0 con 0x7f2fd40f67d0 2024-01-18T07:36:49.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.367+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2fc801d420 con 0x7f2fd40f67d0 2024-01-18T07:36:49.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.367+0000 7f2fc5ffb700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 0x7f2fc005e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:49.370 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.368+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2fc8077f70 con 0x7f2fd40f67d0 2024-01-18T07:36:49.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.368+0000 7f2fd8d05700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 0x7f2fc005e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:49.371 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.368+0000 7f2fd8d05700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 0x7f2fc005e7b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2fd0007fd0 tx=0x7f2fd0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:49.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.371+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fc804afa0 con 0x7f2fd40f67d0 2024-01-18T07:36:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:49.032650+0000 mgr.smithi084.sihbqg (mgr.14162) 580 : cluster [DBG] pgmap v543: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:49.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.580+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2fd4027a00 con 0x7f2fd40f67d0 2024-01-18T07:36:49.582 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.580+0000 7f2fc5ffb700 1 -- 172.21.15.84:0/1087387881 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f2fc804e5c0 con 0x7f2fd40f67d0 2024-01-18T07:36:49.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 msgr2=0x7f2fc005e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 0x7f2fc005e7b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2fd0007fd0 tx=0x7f2fd0009040).stop 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 msgr2=0x7f2fd41070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f2fc8009f50 tx=0x7f2fc8009b30).stop 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 shutdown_connections 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f2fd40f67d0 0x7f2fd41070b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2fc005c2f0 0x7f2fc005e7b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 --2- 172.21.15.84:0/1087387881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2fd40f7a70 0x7f2fd41075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 >> 172.21.15.84:0/1087387881 conn(0x7f2fd40f1f50 msgr2=0x7f2fd40fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:49.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 shutdown_connections 2024-01-18T07:36:49.585 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:49.582+0000 7f2fdb76a700 1 -- 172.21.15.84:0/1087387881 wait complete. 2024-01-18T07:36:49.999 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:50.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:49.582566+0000 mon.smithi084 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.84:0/1087387881' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:50.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:49.582566+0000 mon.smithi084 (mon.0) 536 : audit [DBG] from='client.? 172.21.15.84:0/1087387881' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:51.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:51.033130+0000 mgr.smithi084.sihbqg (mgr.14162) 581 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:51.033130+0000 mgr.smithi084.sihbqg (mgr.14162) 581 : cluster [DBG] pgmap v544: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:53.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:53.235 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:53.373 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:53.033557+0000 mgr.smithi084.sihbqg (mgr.14162) 582 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:53.033557+0000 mgr.smithi084.sihbqg (mgr.14162) 582 : cluster [DBG] pgmap v545: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:54.251 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 -- 172.21.15.84:0/4247476353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 msgr2=0x7ff3cc0f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 --2- 172.21.15.84:0/4247476353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc0f7d10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7ff3bc004660 tx=0x7ff3bc009b30).stop 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 -- 172.21.15.84:0/4247476353 shutdown_connections 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 --2- 172.21.15.84:0/4247476353 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc0f7d10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 --2- 172.21.15.84:0/4247476353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 0x7ff3cc0f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.247+0000 7ff3d305d700 1 -- 172.21.15.84:0/4247476353 >> 172.21.15.84:0/4247476353 conn(0x7ff3cc0f1d70 msgr2=0x7ff3cc0f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 -- 172.21.15.84:0/4247476353 shutdown_connections 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 -- 172.21.15.84:0/4247476353 wait complete. 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 Processor -- start 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 -- start start 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 0x7ff3cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:54.252 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc06a380 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d305d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc06a4c0 con 0x7ff3cc0f6610 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d185a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d185a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59342/0 (socket says 172.21.15.84:59342) 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d185a700 1 -- 172.21.15.84:0/3371156593 learned_addr learned my addr 172.21.15.84:0/3371156593 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.248+0000 7ff3d205b700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 0x7ff3cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d185a700 1 -- 172.21.15.84:0/3371156593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 msgr2=0x7ff3cc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d185a700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 0x7ff3cc06b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d185a700 1 -- 172.21.15.84:0/3371156593 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc005040 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d185a700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7ff3bc009830 tx=0x7ff3bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3bc018070 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3bc013410 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc06a740 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3bc01c400 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.249+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc06aca0 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.250+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc045ad0 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.250+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff3bc01d420 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.253 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.250+0000 7ff3c2ffd700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 0x7ff3b805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:54.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.250+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff3bc077930 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.251+0000 7ff3d205b700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 0x7ff3b805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:54.254 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.251+0000 7ff3d205b700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 0x7ff3b805e770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff3cc06b730 tx=0x7ff3c8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:54.255 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.254+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3bc04a9b0 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.461+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3cc027a00 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.461 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.461+0000 7ff3c2ffd700 1 -- 172.21.15.84:0/3371156593 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff3bc04dfd0 con 0x7ff3cc0f78b0 2024-01-18T07:36:54.461 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 msgr2=0x7ff3b805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 0x7ff3b805e770 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff3cc06b730 tx=0x7ff3c8009040).stop 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 msgr2=0x7ff3cc069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7ff3bc009830 tx=0x7ff3bc009b30).stop 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 shutdown_connections 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff3b805c2b0 0x7ff3b805e770 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.463 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff3cc0f6610 0x7ff3cc06b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 --2- 172.21.15.84:0/3371156593 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3cc0f78b0 0x7ff3cc069e20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.462+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 >> 172.21.15.84:0/3371156593 conn(0x7ff3cc0f1d70 msgr2=0x7ff3cc0fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.463+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 shutdown_connections 2024-01-18T07:36:54.464 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:54.463+0000 7ff3d305d700 1 -- 172.21.15.84:0/3371156593 wait complete. 2024-01-18T07:36:54.812 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:54 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:54.462000+0000 mon.smithi084 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.84:0/3371156593' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:54.870 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:36:54.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:54 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:54.462000+0000 mon.smithi084 (mon.0) 537 : audit [DBG] from='client.? 172.21.15.84:0/3371156593' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:36:55.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:55.033897+0000 mgr.smithi084.sihbqg (mgr.14162) 583 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:55.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:55.033897+0000 mgr.smithi084.sihbqg (mgr.14162) 583 : cluster [DBG] pgmap v546: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:57.034244+0000 mgr.smithi084.sihbqg (mgr.14162) 584 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:57.034244+0000 mgr.smithi084.sihbqg (mgr.14162) 584 : cluster [DBG] pgmap v547: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:57.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:36:58.105 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 -- 172.21.15.84:0/1726104328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f7a60 msgr2=0x7f09c00f7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/1726104328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f7a60 0x7f09c00f7ec0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f09b4004660 tx=0x7f09b4009b30).stop 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 -- 172.21.15.84:0/1726104328 shutdown_connections 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/1726104328 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f7a60 0x7f09c00f7ec0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/1726104328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f67c0 0x7f09c00f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 -- 172.21.15.84:0/1726104328 >> 172.21.15.84:0/1726104328 conn(0x7f09c00f1f20 msgr2=0x7f09c00f4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 -- 172.21.15.84:0/1726104328 shutdown_connections 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.026+0000 7f09c5ee0700 1 -- 172.21.15.84:0/1726104328 wait complete. 2024-01-18T07:36:59.029 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 Processor -- start 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 -- start start 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f7a60 0x7f09c0069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c006a380 con 0x7f09c00f67c0 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09c5ee0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c006a4c0 con 0x7f09c00f7a60 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09bf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09bf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:48986/0 (socket says 172.21.15.84:48986) 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09bf7fe700 1 -- 172.21.15.84:0/2887566840 learned_addr learned my addr 172.21.15.84:0/2887566840 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.027+0000 7f09beffd700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f7a60 0x7f09c0069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09bf7fe700 1 -- 172.21.15.84:0/2887566840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f7a60 msgr2=0x7f09c0069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09bf7fe700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f7a60 0x7f09c0069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09bf7fe700 1 -- 172.21.15.84:0/2887566840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4005040 con 0x7f09c00f67c0 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09bf7fe700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f09b000c790 tx=0x7f09b0007d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:59.030 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b0013630 con 0x7f09c00f67c0 2024-01-18T07:36:59.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09b0012860 con 0x7f09c00f67c0 2024-01-18T07:36:59.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f09b00027e0 con 0x7f09c00f67c0 2024-01-18T07:36:59.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c006a7a0 con 0x7f09c00f67c0 2024-01-18T07:36:59.031 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.028+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c010f0e0 con 0x7f09c00f67c0 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.029+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c0045ad0 con 0x7f09c00f67c0 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.029+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f09b0010070 con 0x7f09c00f67c0 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.030+0000 7f09a7fff700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 0x7f09a805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.030+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f09b0075c40 con 0x7f09c00f67c0 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.030+0000 7f09beffd700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 0x7f09a805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:36:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.030+0000 7f09beffd700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 0x7f09a805e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f09b4004630 tx=0x7f09b4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:36:59.034 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.034+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09b0048be0 con 0x7f09c00f67c0 2024-01-18T07:36:59.131 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:36:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:36:59.034524+0000 mgr.smithi084.sihbqg (mgr.14162) 585 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:59.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.242+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f09c00feff0 con 0x7f09c00f67c0 2024-01-18T07:36:59.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.242+0000 7f09a7fff700 1 -- 172.21.15.84:0/2887566840 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f09b004c200 con 0x7f09c00f67c0 2024-01-18T07:36:59.243 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:36:59.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 msgr2=0x7f09a805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 0x7f09a805e760 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f09b4004630 tx=0x7f09b4009b30).stop 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 msgr2=0x7f09c006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f09b000c790 tx=0x7f09b0007d90).stop 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 shutdown_connections 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f09c00f67c0 0x7f09c006b9a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f09a805c2a0 0x7f09a805e760 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 --2- 172.21.15.84:0/2887566840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f09c00f7a60 0x7f09c0069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 >> 172.21.15.84:0/2887566840 conn(0x7f09c00f1f20 msgr2=0x7f09c00fb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 shutdown_connections 2024-01-18T07:36:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:36:59.244+0000 7f09c5ee0700 1 -- 172.21.15.84:0/2887566840 wait complete. 2024-01-18T07:36:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:36:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:36:59.034524+0000 mgr.smithi084.sihbqg (mgr.14162) 585 : cluster [DBG] pgmap v548: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:36:59.620 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:00.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:36:59.243559+0000 mon.smithi084 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.84:0/2887566840' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:00.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:36:59.243559+0000 mon.smithi084 (mon.0) 538 : audit [DBG] from='client.? 172.21.15.84:0/2887566840' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:01.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:01.034871+0000 mgr.smithi084.sihbqg (mgr.14162) 586 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:01.034871+0000 mgr.smithi084.sihbqg (mgr.14162) 586 : cluster [DBG] pgmap v549: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:02.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:02.857 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:03.332 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:03.035182+0000 mgr.smithi084.sihbqg (mgr.14162) 587 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:03.035182+0000 mgr.smithi084.sihbqg (mgr.14162) 587 : cluster [DBG] pgmap v550: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:03.811 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1793113805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 msgr2=0x7fe4e00f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1793113805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e00f6de0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe4dc004660 tx=0x7fe4dc009b30).stop 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1793113805 shutdown_connections 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1793113805 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e00f6de0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1793113805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 0x7fe4e00f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1793113805 >> 172.21.15.84:0/1793113805 conn(0x7fe4e00f0e20 msgr2=0x7fe4e00f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1793113805 shutdown_connections 2024-01-18T07:37:03.812 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.811+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1793113805 wait complete. 2024-01-18T07:37:03.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 Processor -- start 2024-01-18T07:37:03.813 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 -- start start 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 0x7fe4e0109df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e00609d0 con 0x7fe4e00f6980 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e7a26700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0060b10 con 0x7fe4e00f56e0 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:49012/0 (socket says 172.21.15.84:49012) 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 -- 172.21.15.84:0/1152163197 learned_addr learned my addr 172.21.15.84:0/1152163197 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e57c2700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 0x7fe4e0109df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 -- 172.21.15.84:0/1152163197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 msgr2=0x7fe4e0109df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 0x7fe4e0109df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 -- 172.21.15.84:0/1152163197 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc005040 con 0x7fe4e00f6980 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.812+0000 7fe4e4fc1700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fe4dc009830 tx=0x7fe4dc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.813+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4dc018070 con 0x7fe4e00f6980 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.813+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e010c8f0 con 0x7fe4e00f6980 2024-01-18T07:37:03.814 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.813+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4dc013410 con 0x7fe4e00f6980 2024-01-18T07:37:03.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.813+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4dc01c620 con 0x7fe4e00f6980 2024-01-18T07:37:03.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.813+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e010ce20 con 0x7fe4e00f6980 2024-01-18T07:37:03.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.814+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe4dc01c8e0 con 0x7fe4e00f6980 2024-01-18T07:37:03.815 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.814+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0059100 con 0x7fe4e00f6980 2024-01-18T07:37:03.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.814+0000 7fe4d5ffb700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 0x7fe4cc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:03.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.815+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe4dc077b20 con 0x7fe4e00f6980 2024-01-18T07:37:03.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.815+0000 7fe4e57c2700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 0x7fe4cc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:03.816 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.815+0000 7fe4e57c2700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 0x7fe4cc05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe4e00605c0 tx=0x7fe4d0008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:03.819 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:03.819+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4dc04aa00 con 0x7fe4e00f6980 2024-01-18T07:37:04.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.024+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4e0045ad0 con 0x7fe4e00f6980 2024-01-18T07:37:04.024 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.024+0000 7fe4d5ffb700 1 -- 172.21.15.84:0/1152163197 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fe4dc04e020 con 0x7fe4e00f6980 2024-01-18T07:37:04.025 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 msgr2=0x7fe4cc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 0x7fe4cc05e760 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fe4e00605c0 tx=0x7fe4d0008040).stop 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 msgr2=0x7fe4e010c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fe4dc009830 tx=0x7fe4dc009b30).stop 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 shutdown_connections 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe4cc05c2a0 0x7fe4cc05e760 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4e00f56e0 0x7fe4e0109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 --2- 172.21.15.84:0/1152163197 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe4e00f6980 0x7fe4e010c2f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:04.027 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 >> 172.21.15.84:0/1152163197 conn(0x7fe4e00f0e20 msgr2=0x7fe4e00fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:04.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 shutdown_connections 2024-01-18T07:37:04.028 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:04.026+0000 7fe4e7a26700 1 -- 172.21.15.84:0/1152163197 wait complete. 2024-01-18T07:37:04.343 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:04 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:04.025308+0000 mon.smithi084 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.84:0/1152163197' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:04.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:04 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:04.025308+0000 mon.smithi084 (mon.0) 539 : audit [DBG] from='client.? 172.21.15.84:0/1152163197' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:04.444 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:05.035592+0000 mgr.smithi084.sihbqg (mgr.14162) 588 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:05.035592+0000 mgr.smithi084.sihbqg (mgr.14162) 588 : cluster [DBG] pgmap v551: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:07.035923+0000 mgr.smithi084.sihbqg (mgr.14162) 589 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:07.035923+0000 mgr.smithi084.sihbqg (mgr.14162) 589 : cluster [DBG] pgmap v552: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:07.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:07.680 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 -- 172.21.15.84:0/594115496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f6960 msgr2=0x7fd73c0f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 --2- 172.21.15.84:0/594115496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f6960 0x7fd73c0f6dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fd724004660 tx=0x7fd724009b30).stop 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 -- 172.21.15.84:0/594115496 shutdown_connections 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 --2- 172.21.15.84:0/594115496 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f6960 0x7fd73c0f6dc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 --2- 172.21.15.84:0/594115496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f56c0 0x7fd73c0f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 -- 172.21.15.84:0/594115496 >> 172.21.15.84:0/594115496 conn(0x7fd73c0f0e20 msgr2=0x7fd73c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 -- 172.21.15.84:0/594115496 shutdown_connections 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.667+0000 7fd74213b700 1 -- 172.21.15.84:0/594115496 wait complete. 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 Processor -- start 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 -- start start 2024-01-18T07:37:08.670 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 0x7fd73c10c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c060940 con 0x7fd73c0f56c0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd74213b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73c060a80 con 0x7fd73c0f6960 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd73b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd73b7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41440/0 (socket says 172.21.15.84:41440) 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.668+0000 7fd73b7fe700 1 -- 172.21.15.84:0/424487655 learned_addr learned my addr 172.21.15.84:0/424487655 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd733fff700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 0x7fd73c10c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd73b7fe700 1 -- 172.21.15.84:0/424487655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 msgr2=0x7fd73c10c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd73b7fe700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 0x7fd73c10c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd73b7fe700 1 -- 172.21.15.84:0/424487655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd724005040 con 0x7fd73c0f56c0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd733fff700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 0x7fd73c10c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd73b7fe700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd72c002700 tx=0x7fd72c009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd72c009320 con 0x7fd73c0f56c0 2024-01-18T07:37:08.671 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd72c00f040 con 0x7fd73c0f56c0 2024-01-18T07:37:08.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd72c014900 con 0x7fd73c0f56c0 2024-01-18T07:37:08.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73c10c890 con 0x7fd73c0f56c0 2024-01-18T07:37:08.672 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.669+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd73c10ce50 con 0x7fd73c0f56c0 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.670+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73c045ad0 con 0x7fd73c0f56c0 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.670+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fd72c009480 con 0x7fd73c0f56c0 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.671+0000 7fd738ff9700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 0x7fd72805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.671+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fd72c075db0 con 0x7fd73c0f56c0 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.671+0000 7fd733fff700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 0x7fd72805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:08.673 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.671+0000 7fd733fff700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 0x7fd72805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fd724009d60 tx=0x7fd724009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:08.675 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.675+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd72c048e30 con 0x7fd73c0f56c0 2024-01-18T07:37:08.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.881+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd73c0fdf50 con 0x7fd73c0f56c0 2024-01-18T07:37:08.882 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.882+0000 7fd738ff9700 1 -- 172.21.15.84:0/424487655 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fd72c04c450 con 0x7fd73c0f56c0 2024-01-18T07:37:08.883 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 msgr2=0x7fd72805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 0x7fd72805e760 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fd724009d60 tx=0x7fd724009b30).stop 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 msgr2=0x7fd73c109de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd72c002700 tx=0x7fd72c009d80).stop 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 shutdown_connections 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fd73c0f56c0 0x7fd73c109de0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.884 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fd72805c2a0 0x7fd72805e760 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 --2- 172.21.15.84:0/424487655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd73c0f6960 0x7fd73c10c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:08.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.883+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 >> 172.21.15.84:0/424487655 conn(0x7fd73c0f0e20 msgr2=0x7fd73c0fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:08.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.884+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 shutdown_connections 2024-01-18T07:37:08.885 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:08.884+0000 7fd74213b700 1 -- 172.21.15.84:0/424487655 wait complete. 2024-01-18T07:37:09.175 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:08 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:08.882896+0000 mon.smithi084 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.84:0/424487655' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:09.259 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:08 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:08.882896+0000 mon.smithi084 (mon.0) 540 : audit [DBG] from='client.? 172.21.15.84:0/424487655' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:10.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:09.036232+0000 mgr.smithi084.sihbqg (mgr.14162) 590 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:10.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:09.036232+0000 mgr.smithi084.sihbqg (mgr.14162) 590 : cluster [DBG] pgmap v553: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:11.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:11.036592+0000 mgr.smithi084.sihbqg (mgr.14162) 591 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:11.036592+0000 mgr.smithi084.sihbqg (mgr.14162) 591 : cluster [DBG] pgmap v554: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:12.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:12.494 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:13.221 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:13.037112+0000 mgr.smithi084.sihbqg (mgr.14162) 592 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:13.037112+0000 mgr.smithi084.sihbqg (mgr.14162) 592 : cluster [DBG] pgmap v555: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:13.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 -- 172.21.15.84:0/569760446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 msgr2=0x7f84940f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:13.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 --2- 172.21.15.84:0/569760446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84940f93a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8484004660 tx=0x7f8484009b30).stop 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 -- 172.21.15.84:0/569760446 shutdown_connections 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 --2- 172.21.15.84:0/569760446 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 0x7f84940fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 --2- 172.21.15.84:0/569760446 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84940f93a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 -- 172.21.15.84:0/569760446 >> 172.21.15.84:0/569760446 conn(0x7f84940f0e20 msgr2=0x7f84940f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 -- 172.21.15.84:0/569760446 shutdown_connections 2024-01-18T07:37:13.427 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.426+0000 7f849a16a700 1 -- 172.21.15.84:0/569760446 wait complete. 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 Processor -- start 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 -- start start 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 0x7f84941075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494107b50 con 0x7f8494060870 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f849a16a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494107c90 con 0x7f84940f9920 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:13.429 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:41458/0 (socket says 172.21.15.84:41458) 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 -- 172.21.15.84:0/3447875189 learned_addr learned my addr 172.21.15.84:0/3447875189 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 -- 172.21.15.84:0/3447875189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 msgr2=0x7f84941075a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f8492ffd700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 0x7f84941075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 0x7f84941075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 -- 172.21.15.84:0/3447875189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484005040 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.427+0000 7f84937fe700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f8484009830 tx=0x7f8484009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8484018070 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8484013410 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84941085d0 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f848401c620 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8494108b30 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.428+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8494045ad0 con 0x7f8494060870 2024-01-18T07:37:13.430 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.429+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f848401c8e0 con 0x7f8494060870 2024-01-18T07:37:13.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.429+0000 7f847bfff700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 0x7f847c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:13.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.429+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8484077a80 con 0x7f8494060870 2024-01-18T07:37:13.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.429+0000 7f8492ffd700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 0x7f847c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:13.431 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.430+0000 7f8492ffd700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 0x7f847c05e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f84880041d0 tx=0x7f8488009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:13.433 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.433+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f848404aa20 con 0x7f8494060870 2024-01-18T07:37:13.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.640+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f84940ff0b0 con 0x7f8494060870 2024-01-18T07:37:13.640 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.640+0000 7f847bfff700 1 -- 172.21.15.84:0/3447875189 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f848404e040 con 0x7f8494060870 2024-01-18T07:37:13.640 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 msgr2=0x7f847c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 0x7f847c05e760 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f84880041d0 tx=0x7f8488009040).stop 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 msgr2=0x7f84941070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f8484009830 tx=0x7f8484009b30).stop 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 shutdown_connections 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8494060870 0x7f84941070b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f847c05c2a0 0x7f847c05e760 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.642 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 --2- 172.21.15.84:0/3447875189 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84940f9920 0x7f84941075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:13.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.641+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 >> 172.21.15.84:0/3447875189 conn(0x7f84940f0e20 msgr2=0x7f84940f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:13.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.642+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 shutdown_connections 2024-01-18T07:37:13.643 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:13.642+0000 7f849a16a700 1 -- 172.21.15.84:0/3447875189 wait complete. 2024-01-18T07:37:14.009 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:14.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:14 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:13.641013+0000 mon.smithi084 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.84:0/3447875189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:14.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:14 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:13.641013+0000 mon.smithi084 (mon.0) 541 : audit [DBG] from='client.? 172.21.15.84:0/3447875189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:15.037443+0000 mgr.smithi084.sihbqg (mgr.14162) 593 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:15.037443+0000 mgr.smithi084.sihbqg (mgr.14162) 593 : cluster [DBG] pgmap v556: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:17.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:17.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:17.376 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:17.037836+0000 mgr.smithi084.sihbqg (mgr.14162) 594 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:17.037836+0000 mgr.smithi084.sihbqg (mgr.14162) 594 : cluster [DBG] pgmap v557: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 -- 172.21.15.84:0/2321438742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c4060870 msgr2=0x7fe3c40f93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2321438742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c4060870 0x7fe3c40f93a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe3bc0099e0 tx=0x7fe3bc004d10).stop 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 -- 172.21.15.84:0/2321438742 shutdown_connections 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2321438742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c40f9920 0x7fe3c40fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2321438742 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c4060870 0x7fe3c40f93a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 -- 172.21.15.84:0/2321438742 >> 172.21.15.84:0/2321438742 conn(0x7fe3c40f0e20 msgr2=0x7fe3c40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:18.227 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 -- 172.21.15.84:0/2321438742 shutdown_connections 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 -- 172.21.15.84:0/2321438742 wait complete. 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.225+0000 7fe3ca102700 1 Processor -- start 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 -- start start 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c4060870 0x7fe3c4107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c4107d40 con 0x7fe3c40f9920 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c41085f0 con 0x7fe3c4060870 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43654/0 (socket says 172.21.15.84:43654) 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 -- 172.21.15.84:0/2825455757 learned_addr learned my addr 172.21.15.84:0/2825455757 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c37fe700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c4060870 0x7fe3c4107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 -- 172.21.15.84:0/2825455757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c4060870 msgr2=0x7fe3c4107210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:18.228 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c4060870 0x7fe3c4107210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 -- 172.21.15.84:0/2825455757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc009640 con 0x7fe3c40f9920 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3c2ffd700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fe3ac0093f0 tx=0x7fe3ac006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3ac013640 con 0x7fe3c40f9920 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c41088d0 con 0x7fe3c40f9920 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3ac013c50 con 0x7fe3c40f9920 2024-01-18T07:37:18.229 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.226+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe3ac012c80 con 0x7fe3c40f9920 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.227+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c4108e10 con 0x7fe3c40f9920 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.227+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c4059100 con 0x7fe3c40f9920 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.228+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fe3ac01a060 con 0x7fe3c40f9920 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.228+0000 7fe3bbfff700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 0x7fe3b005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.228+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fe3ac075b90 con 0x7fe3c40f9920 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.228+0000 7fe3c37fe700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 0x7fe3b005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:18.230 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.229+0000 7fe3c37fe700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 0x7fe3b005e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3bc003380 tx=0x7fe3bc004d10).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:18.232 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.232+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3ac048b90 con 0x7fe3c40f9920 2024-01-18T07:37:18.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.440+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3c40feff0 con 0x7fe3c40f9920 2024-01-18T07:37:18.440 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.440+0000 7fe3bbfff700 1 -- 172.21.15.84:0/2825455757 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fe3ac04c1b0 con 0x7fe3c40f9920 2024-01-18T07:37:18.440 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.441+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 msgr2=0x7fe3b005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.441+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 0x7fe3b005e760 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fe3bc003380 tx=0x7fe3bc004d10).stop 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 msgr2=0x7fe3c4107700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fe3ac0093f0 tx=0x7fe3ac006ec0).stop 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 shutdown_connections 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fe3b005c2a0 0x7fe3b005e760 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3c4060870 0x7fe3c4107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:18.442 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 --2- 172.21.15.84:0/2825455757 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fe3c40f9920 0x7fe3c4107700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:18.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 >> 172.21.15.84:0/2825455757 conn(0x7fe3c40f0e20 msgr2=0x7fe3c40f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:18.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 shutdown_connections 2024-01-18T07:37:18.443 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:18.442+0000 7fe3ca102700 1 -- 172.21.15.84:0/2825455757 wait complete. 2024-01-18T07:37:18.773 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:18.441131+0000 mon.smithi084 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.84:0/2825455757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:18.867 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:18.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:18.441131+0000 mon.smithi084 (mon.0) 542 : audit [DBG] from='client.? 172.21.15.84:0/2825455757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:19.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:19.038302+0000 mgr.smithi084.sihbqg (mgr.14162) 595 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:19.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:19.038302+0000 mgr.smithi084.sihbqg (mgr.14162) 595 : cluster [DBG] pgmap v558: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:21.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:21.038697+0000 mgr.smithi084.sihbqg (mgr.14162) 596 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:21.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:21.038697+0000 mgr.smithi084.sihbqg (mgr.14162) 596 : cluster [DBG] pgmap v559: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:21.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:22.102 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 -- 172.21.15.84:0/1672742717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 msgr2=0x7f36800f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 --2- 172.21.15.84:0/1672742717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36800f5ae0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3670004660 tx=0x7f3670009b30).stop 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 -- 172.21.15.84:0/1672742717 shutdown_connections 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 --2- 172.21.15.84:0/1672742717 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36800f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 --2- 172.21.15.84:0/1672742717 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36800f5ae0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 -- 172.21.15.84:0/1672742717 >> 172.21.15.84:0/1672742717 conn(0x7f36800f0e20 msgr2=0x7f36800f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 -- 172.21.15.84:0/1672742717 shutdown_connections 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.106+0000 7f3688114700 1 -- 172.21.15.84:0/1672742717 wait complete. 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 Processor -- start 2024-01-18T07:37:23.109 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 -- start start 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36801097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3680109e10 con 0x7f36800f56c0 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368010a6c0 con 0x7f36800f6960 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43662/0 (socket says 172.21.15.84:43662) 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 -- 172.21.15.84:0/251190402 learned_addr learned my addr 172.21.15.84:0/251190402 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f36856af700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36801097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 -- 172.21.15.84:0/251190402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 msgr2=0x7f36801097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36801097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 -- 172.21.15.84:0/251190402 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670005040 con 0x7f36800f56c0 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f36856af700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36801097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3685eb0700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3670009f50 tx=0x7f3670009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:23.110 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3670018070 con 0x7f36800f56c0 2024-01-18T07:37:23.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3670013410 con 0x7f36800f56c0 2024-01-18T07:37:23.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368010a940 con 0x7f36800f56c0 2024-01-18T07:37:23.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.107+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f367001c620 con 0x7f36800f56c0 2024-01-18T07:37:23.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.108+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368010aea0 con 0x7f36800f56c0 2024-01-18T07:37:23.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.108+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3680059100 con 0x7f36800f56c0 2024-01-18T07:37:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.109+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f367001d420 con 0x7f36800f56c0 2024-01-18T07:37:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.109+0000 7f36767fc700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 0x7f366c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.109+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3670077eb0 con 0x7f36800f56c0 2024-01-18T07:37:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.109+0000 7f36856af700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 0x7f366c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:23.112 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.110+0000 7f36856af700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 0x7f366c05e760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f367c00a7e0 tx=0x7f367c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:23.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.113+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f367004afa0 con 0x7f36800f56c0 2024-01-18T07:37:23.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.320+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3680045ad0 con 0x7f36800f56c0 2024-01-18T07:37:23.321 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.321+0000 7f36767fc700 1 -- 172.21.15.84:0/251190402 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f367004e5c0 con 0x7f36800f56c0 2024-01-18T07:37:23.321 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.322+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 msgr2=0x7f366c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.322+0000 7f3688114700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 0x7f366c05e760 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f367c00a7e0 tx=0x7f367c008040).stop 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 msgr2=0x7f36801092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3670009f50 tx=0x7f3670009b30).stop 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 shutdown_connections 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f36800f56c0 0x7f36801092e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:23.323 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f366c05c2a0 0x7f366c05e760 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:23.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 --2- 172.21.15.84:0/251190402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36800f6960 0x7f36801097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:23.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 >> 172.21.15.84:0/251190402 conn(0x7f36800f0e20 msgr2=0x7f36800603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:23.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 shutdown_connections 2024-01-18T07:37:23.324 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:23.323+0000 7f3688114700 1 -- 172.21.15.84:0/251190402 wait complete. 2024-01-18T07:37:23.342 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:23.038984+0000 mgr.smithi084.sihbqg (mgr.14162) 597 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:23.038984+0000 mgr.smithi084.sihbqg (mgr.14162) 597 : cluster [DBG] pgmap v560: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:23.698 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:24.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:23.321998+0000 mon.smithi084 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.84:0/251190402' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:24.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:23.321998+0000 mon.smithi084 (mon.0) 543 : audit [DBG] from='client.? 172.21.15.84:0/251190402' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:25.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:25.039282+0000 mgr.smithi084.sihbqg (mgr.14162) 598 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:25.039282+0000 mgr.smithi084.sihbqg (mgr.14162) 598 : cluster [DBG] pgmap v561: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:26.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:26.934 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:27.039662+0000 mgr.smithi084.sihbqg (mgr.14162) 599 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:27.039662+0000 mgr.smithi084.sihbqg (mgr.14162) 599 : cluster [DBG] pgmap v562: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 -- 172.21.15.84:0/57178967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f78b0 msgr2=0x7f74840f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 --2- 172.21.15.84:0/57178967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f78b0 0x7f74840f7d10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7474004660 tx=0x7f7474009b30).stop 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 -- 172.21.15.84:0/57178967 shutdown_connections 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 --2- 172.21.15.84:0/57178967 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f78b0 0x7f74840f7d10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 --2- 172.21.15.84:0/57178967 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f6610 0x7f74840f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 -- 172.21.15.84:0/57178967 >> 172.21.15.84:0/57178967 conn(0x7f74840f1d70 msgr2=0x7f74840f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.896+0000 7f748a7e1700 1 -- 172.21.15.84:0/57178967 shutdown_connections 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 -- 172.21.15.84:0/57178967 wait complete. 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 Processor -- start 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 -- start start 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:27.899 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f78b0 0x7f74841073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7484107970 con 0x7f74840f6610 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f748a7e1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7484107ab0 con 0x7f74840f78b0 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f74897df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f74897df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43308/0 (socket says 172.21.15.84:43308) 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f74897df700 1 -- 172.21.15.84:0/474888343 learned_addr learned my addr 172.21.15.84:0/474888343 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.897+0000 7f7488fde700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f78b0 0x7f74841073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f74897df700 1 -- 172.21.15.84:0/474888343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f78b0 msgr2=0x7f74841073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f74897df700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f78b0 0x7f74841073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f74897df700 1 -- 172.21.15.84:0/474888343 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474005040 con 0x7f74840f6610 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f74897df700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f748000c530 tx=0x7f7480007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f748000f650 con 0x7f74840f6610 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7484108450 con 0x7f74840f6610 2024-01-18T07:37:27.900 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f748000fc60 con 0x7f74840f6610 2024-01-18T07:37:27.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7480012c60 con 0x7f74840f6610 2024-01-18T07:37:27.901 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.898+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7484108a10 con 0x7f74840f6610 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.899+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7484045ad0 con 0x7f74840f6610 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.899+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f7480019030 con 0x7f74840f6610 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.900+0000 7f7479ffb700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 0x7f747005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.900+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7480075720 con 0x7f74840f6610 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.900+0000 7f7488fde700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 0x7f747005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:27.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.900+0000 7f7488fde700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 0x7f747005e770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7474004630 tx=0x7f7474009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:27.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:27.904+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74800462b0 con 0x7f74840f6610 2024-01-18T07:37:28.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.110+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74840fef20 con 0x7f74840f6610 2024-01-18T07:37:28.111 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.111+0000 7f7479ffb700 1 -- 172.21.15.84:0/474888343 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f7480017070 con 0x7f74840f6610 2024-01-18T07:37:28.111 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 msgr2=0x7f747005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 0x7f747005e770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7474004630 tx=0x7f7474009b30).stop 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 msgr2=0x7f7484106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f748000c530 tx=0x7f7480007e10).stop 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 shutdown_connections 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f74840f6610 0x7f7484106ed0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f747005c2b0 0x7f747005e770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 --2- 172.21.15.84:0/474888343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74840f78b0 0x7f74841073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 >> 172.21.15.84:0/474888343 conn(0x7f74840f1d70 msgr2=0x7f74840fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 shutdown_connections 2024-01-18T07:37:28.113 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:28.112+0000 7f748a7e1700 1 -- 172.21.15.84:0/474888343 wait complete. 2024-01-18T07:37:28.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:28 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:28.111730+0000 mon.smithi084 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.84:0/474888343' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:28.438 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:28 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:28.111730+0000 mon.smithi084 (mon.0) 544 : audit [DBG] from='client.? 172.21.15.84:0/474888343' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:28.497 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:29.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:29.040134+0000 mgr.smithi084.sihbqg (mgr.14162) 600 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:29.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:29.040134+0000 mgr.smithi084.sihbqg (mgr.14162) 600 : cluster [DBG] pgmap v563: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:31.040520+0000 mgr.smithi084.sihbqg (mgr.14162) 601 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:31.040520+0000 mgr.smithi084.sihbqg (mgr.14162) 601 : cluster [DBG] pgmap v564: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:31.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:31.732 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 -- 172.21.15.84:0/219115192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 msgr2=0x7fb1700f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 --2- 172.21.15.84:0/219115192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1700f9400 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb160004660 tx=0x7fb160009b30).stop 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 -- 172.21.15.84:0/219115192 shutdown_connections 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 --2- 172.21.15.84:0/219115192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1700f9980 0x7fb1700fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 --2- 172.21.15.84:0/219115192 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1700f9400 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 -- 172.21.15.84:0/219115192 >> 172.21.15.84:0/219115192 conn(0x7fb1700f0e20 msgr2=0x7fb1700f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 -- 172.21.15.84:0/219115192 shutdown_connections 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 -- 172.21.15.84:0/219115192 wait complete. 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.727+0000 7fb1763b8700 1 Processor -- start 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 -- start start 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1700f9980 0x7fb1701075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb170107c10 con 0x7fb170060810 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701084c0 con 0x7fb1700f9980 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:43312/0 (socket says 172.21.15.84:43312) 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 -- 172.21.15.84:0/3110713103 learned_addr learned my addr 172.21.15.84:0/3110713103 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 -- 172.21.15.84:0/3110713103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1700f9980 msgr2=0x7fb1701075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1700f9980 0x7fb1701075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 -- 172.21.15.84:0/3110713103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160005040 con 0x7fb170060810 2024-01-18T07:37:32.730 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16ffff700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fb160009830 tx=0x7fb160009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:32.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb160018070 con 0x7fb170060810 2024-01-18T07:37:32.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb170108740 con 0x7fb170060810 2024-01-18T07:37:32.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb160013410 con 0x7fb170060810 2024-01-18T07:37:32.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.728+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb16001c620 con 0x7fb170060810 2024-01-18T07:37:32.731 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.729+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb170108ca0 con 0x7fb170060810 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.729+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb170059100 con 0x7fb170060810 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.730+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fb16001c8e0 con 0x7fb170060810 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.730+0000 7fb16cff9700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 0x7fb15805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.730+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fb160077a60 con 0x7fb170060810 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.730+0000 7fb16f7fe700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 0x7fb15805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:32.732 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.731+0000 7fb16f7fe700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 0x7fb15805e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb1700605e0 tx=0x7fb164006ab0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:32.734 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.734+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb16004aa00 con 0x7fb170060810 2024-01-18T07:37:32.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.941+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb170045ad0 con 0x7fb170060810 2024-01-18T07:37:32.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.942+0000 7fb16cff9700 1 -- 172.21.15.84:0/3110713103 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fb16004e020 con 0x7fb170060810 2024-01-18T07:37:32.942 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 msgr2=0x7fb15805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 0x7fb15805e760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb1700605e0 tx=0x7fb164006ab0).stop 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 msgr2=0x7fb1701070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fb160009830 tx=0x7fb160009b30).stop 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 shutdown_connections 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fb170060810 0x7fb1701070e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fb15805c2a0 0x7fb15805e760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 --2- 172.21.15.84:0/3110713103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1700f9980 0x7fb1701075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:32.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.943+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 >> 172.21.15.84:0/3110713103 conn(0x7fb1700f0e20 msgr2=0x7fb1700f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:32.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.944+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 shutdown_connections 2024-01-18T07:37:32.945 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:32.944+0000 7fb1763b8700 1 -- 172.21.15.84:0/3110713103 wait complete. 2024-01-18T07:37:33.278 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:32.942855+0000 mon.smithi084 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.84:0/3110713103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:33.355 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:32.942855+0000 mon.smithi084 (mon.0) 545 : audit [DBG] from='client.? 172.21.15.84:0/3110713103' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:34.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:33.040849+0000 mgr.smithi084.sihbqg (mgr.14162) 602 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:34.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:33.040849+0000 mgr.smithi084.sihbqg (mgr.14162) 602 : cluster [DBG] pgmap v565: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:35.041200+0000 mgr.smithi084.sihbqg (mgr.14162) 603 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:35.056717+0000 mon.smithi084 (mon.0) 546 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:37:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:35.066995+0000 mon.smithi084 (mon.0) 547 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:37:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:35.041200+0000 mgr.smithi084.sihbqg (mgr.14162) 603 : cluster [DBG] pgmap v566: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:35.056717+0000 mon.smithi084 (mon.0) 546 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:37:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:35.066995+0000 mon.smithi084 (mon.0) 547 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:37:36.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:36.591 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:37.375 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:37.041671+0000 mgr.smithi084.sihbqg (mgr.14162) 604 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:37.041671+0000 mgr.smithi084.sihbqg (mgr.14162) 604 : cluster [DBG] pgmap v567: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:37.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 -- 172.21.15.84:0/2361312869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f6980 msgr2=0x7f98ac0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:37.560 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 --2- 172.21.15.84:0/2361312869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f6980 0x7f98ac0f6de0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f98a8004660 tx=0x7f98a8009b30).stop 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 -- 172.21.15.84:0/2361312869 shutdown_connections 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 --2- 172.21.15.84:0/2361312869 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f6980 0x7f98ac0f6de0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 --2- 172.21.15.84:0/2361312869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f56e0 0x7f98ac0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 -- 172.21.15.84:0/2361312869 >> 172.21.15.84:0/2361312869 conn(0x7f98ac0f0e20 msgr2=0x7f98ac0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 -- 172.21.15.84:0/2361312869 shutdown_connections 2024-01-18T07:37:37.561 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.559+0000 7f98b3b28700 1 -- 172.21.15.84:0/2361312869 wait complete. 2024-01-18T07:37:37.562 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 Processor -- start 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 -- start start 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 0x7f98ac107590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac107b40 con 0x7f98ac0f56e0 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac107c80 con 0x7f98ac0f6980 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58714/0 (socket says 172.21.15.84:58714) 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 -- 172.21.15.84:0/4117171385 learned_addr learned my addr 172.21.15.84:0/4117171385 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b10c3700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 0x7f98ac107590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 -- 172.21.15.84:0/4117171385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 msgr2=0x7f98ac107590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 0x7f98ac107590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 -- 172.21.15.84:0/4117171385 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a8005040 con 0x7f98ac0f56e0 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b10c3700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 0x7f98ac107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b18c4700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f989c00c530 tx=0x7f989c007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:37.563 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f989c00f650 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f989c00fc60 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac108620 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f989c012d20 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.560+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac108be0 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.561+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac045ad0 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.562+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f989c019030 con 0x7f98ac0f56e0 2024-01-18T07:37:37.564 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.562+0000 7f98a27fc700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 0x7f989805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:37.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.562+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f989c0757f0 con 0x7f98ac0f56e0 2024-01-18T07:37:37.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.562+0000 7f98b10c3700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 0x7f989805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:37.565 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.562+0000 7f98b10c3700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 0x7f989805e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f98a8004e30 tx=0x7f98a8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:37.567 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.566+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f989c048840 con 0x7f98ac0f56e0 2024-01-18T07:37:37.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.773+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98ac1092e0 con 0x7f98ac0f56e0 2024-01-18T07:37:37.774 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.773+0000 7f98a27fc700 1 -- 172.21.15.84:0/4117171385 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f989c017080 con 0x7f98ac0f56e0 2024-01-18T07:37:37.774 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 msgr2=0x7f989805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 0x7f989805e760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f98a8004e30 tx=0x7f98a8009b30).stop 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 msgr2=0x7f98ac1070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f989c00c530 tx=0x7f989c007e10).stop 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 shutdown_connections 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f98ac0f56e0 0x7f98ac1070a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f989805c2a0 0x7f989805e760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.776 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 --2- 172.21.15.84:0/4117171385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98ac0f6980 0x7f98ac107590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:37.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 >> 172.21.15.84:0/4117171385 conn(0x7f98ac0f0e20 msgr2=0x7f98ac0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:37.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 shutdown_connections 2024-01-18T07:37:37.777 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:37.775+0000 7f98b3b28700 1 -- 172.21.15.84:0/4117171385 wait complete. 2024-01-18T07:37:38.144 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:38.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:37.775122+0000 mon.smithi084 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.84:0/4117171385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:38.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:38 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:37.775122+0000 mon.smithi084 (mon.0) 548 : audit [DBG] from='client.? 172.21.15.84:0/4117171385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:39.042104+0000 mgr.smithi084.sihbqg (mgr.14162) 605 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:39.042104+0000 mgr.smithi084.sihbqg (mgr.14162) 605 : cluster [DBG] pgmap v568: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:41.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:41.380 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:41.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:41.042482+0000 mgr.smithi084.sihbqg (mgr.14162) 606 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:41.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:41.042482+0000 mgr.smithi084.sihbqg (mgr.14162) 606 : cluster [DBG] pgmap v569: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:42.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 -- 172.21.15.84:0/1049952088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f56c0 msgr2=0x7f3ab40f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:42.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1049952088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f56c0 0x7f3ab40f5ae0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f3aa8004660 tx=0x7f3aa8009b30).stop 2024-01-18T07:37:42.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 -- 172.21.15.84:0/1049952088 shutdown_connections 2024-01-18T07:37:42.372 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1049952088 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f6960 0x7f3ab40f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1049952088 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f56c0 0x7f3ab40f5ae0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 -- 172.21.15.84:0/1049952088 >> 172.21.15.84:0/1049952088 conn(0x7f3ab40f0e20 msgr2=0x7f3ab40f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 -- 172.21.15.84:0/1049952088 shutdown_connections 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.369+0000 7f3abadf3700 1 -- 172.21.15.84:0/1049952088 wait complete. 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 Processor -- start 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 -- start start 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 0x7f3ab4109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab4060940 con 0x7f3ab40f6960 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3abadf3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab4060a80 con 0x7f3ab40f56c0 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:58746/0 (socket says 172.21.15.84:58746) 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 -- 172.21.15.84:0/1308951689 learned_addr learned my addr 172.21.15.84:0/1308951689 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:42.373 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 -- 172.21.15.84:0/1308951689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 msgr2=0x7f3ab4109dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab8b8f700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 0x7f3ab4109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 0x7f3ab4109dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 -- 172.21.15.84:0/1308951689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa8005040 con 0x7f3ab40f6960 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab8b8f700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 0x7f3ab4109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.370+0000 7f3ab3fff700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3aa00026d0 tx=0x7f3aa0009de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.371+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3aa0009320 con 0x7f3ab40f6960 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.371+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab410c880 con 0x7f3ab40f6960 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.371+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3aa000f040 con 0x7f3ab40f6960 2024-01-18T07:37:42.374 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.371+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3aa00147e0 con 0x7f3ab40f6960 2024-01-18T07:37:42.375 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.371+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab410ce40 con 0x7f3ab40f6960 2024-01-18T07:37:42.375 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.372+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab4045ad0 con 0x7f3ab40f6960 2024-01-18T07:37:42.375 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.372+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f3aa00077b0 con 0x7f3ab40f6960 2024-01-18T07:37:42.376 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.372+0000 7f3ab17fa700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 0x7f3aa405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:42.376 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.372+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3aa0076290 con 0x7f3ab40f6960 2024-01-18T07:37:42.376 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.372+0000 7f3ab8b8f700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 0x7f3aa405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:42.376 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.373+0000 7f3ab8b8f700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 0x7f3aa405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3aa8009d60 tx=0x7f3aa8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:42.377 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.376+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3aa0049290 con 0x7f3ab40f6960 2024-01-18T07:37:42.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.580+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ab4027a00 con 0x7f3ab40f6960 2024-01-18T07:37:42.581 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.580+0000 7f3ab17fa700 1 -- 172.21.15.84:0/1308951689 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f3aa004c8b0 con 0x7f3ab40f6960 2024-01-18T07:37:42.582 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:42.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 msgr2=0x7f3aa405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:42.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 0x7f3aa405e760 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f3aa8009d60 tx=0x7f3aa8009b30).stop 2024-01-18T07:37:42.583 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 msgr2=0x7f3ab410c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3aa00026d0 tx=0x7f3aa0009de0).stop 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 shutdown_connections 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f3aa405c2a0 0x7f3aa405e760 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ab40f56c0 0x7f3ab4109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 --2- 172.21.15.84:0/1308951689 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f3ab40f6960 0x7f3ab410c2d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 >> 172.21.15.84:0/1308951689 conn(0x7f3ab40f0e20 msgr2=0x7f3ab40fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 shutdown_connections 2024-01-18T07:37:42.584 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:42.582+0000 7f3abadf3700 1 -- 172.21.15.84:0/1308951689 wait complete. 2024-01-18T07:37:42.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:42 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:42.582247+0000 mon.smithi084 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.84:0/1308951689' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:42.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:42.582247+0000 mon.smithi084 (mon.0) 549 : audit [DBG] from='client.? 172.21.15.84:0/1308951689' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:42.969 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:43.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:43.042756+0000 mgr.smithi084.sihbqg (mgr.14162) 607 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:44.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:43.042756+0000 mgr.smithi084.sihbqg (mgr.14162) 607 : cluster [DBG] pgmap v570: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:45.043143+0000 mgr.smithi084.sihbqg (mgr.14162) 608 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:45.043143+0000 mgr.smithi084.sihbqg (mgr.14162) 608 : cluster [DBG] pgmap v571: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:45.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:46.204 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- 172.21.15.84:0/2012548325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 msgr2=0x7f28c80f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 --2- 172.21.15.84:0/2012548325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c80f5930 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f28c4004660 tx=0x7f28c4009b30).stop 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- 172.21.15.84:0/2012548325 shutdown_connections 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 --2- 172.21.15.84:0/2012548325 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 0x7f28c80f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 --2- 172.21.15.84:0/2012548325 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c80f5930 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- 172.21.15.84:0/2012548325 >> 172.21.15.84:0/2012548325 conn(0x7f28c80f0c70 msgr2=0x7f28c80f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- 172.21.15.84:0/2012548325 shutdown_connections 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- 172.21.15.84:0/2012548325 wait complete. 2024-01-18T07:37:47.193 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 Processor -- start 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- start start 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 0x7f28c81073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8107980 con 0x7f28c80f5510 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cea31700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8107ac0 con 0x7f28c80f67b0 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cda2f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.191+0000 7f28cda2f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54702/0 (socket says 172.21.15.84:54702) 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cda2f700 1 -- 172.21.15.84:0/4222435896 learned_addr learned my addr 172.21.15.84:0/4222435896 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cd22e700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 0x7f28c81073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cda2f700 1 -- 172.21.15.84:0/4222435896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 msgr2=0x7f28c81073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cda2f700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 0x7f28c81073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cda2f700 1 -- 172.21.15.84:0/4222435896 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c4005040 con 0x7f28c80f5510 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cda2f700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f28c4004e30 tx=0x7f28c4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c4018070 con 0x7f28c80f5510 2024-01-18T07:37:47.194 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8108400 con 0x7f28c80f5510 2024-01-18T07:37:47.195 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28c4013410 con 0x7f28c80f5510 2024-01-18T07:37:47.195 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28c401c3c0 con 0x7f28c80f5510 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.192+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c8108960 con 0x7f28c80f5510 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.193+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c8045ad0 con 0x7f28c80f5510 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.193+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f28c401d420 con 0x7f28c80f5510 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.194+0000 7f28be7fc700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 0x7f28b405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.194+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f28c4077820 con 0x7f28c80f5510 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.194+0000 7f28cd22e700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 0x7f28b405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:47.196 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.194+0000 7f28cd22e700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 0x7f28b405e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f28b80096c0 tx=0x7f28b8009be0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:47.198 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.198+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28c404a820 con 0x7f28c80f5510 2024-01-18T07:37:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:47.043576+0000 mgr.smithi084.sihbqg (mgr.14162) 609 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:47.513 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:47.043576+0000 mgr.smithi084.sihbqg (mgr.14162) 609 : cluster [DBG] pgmap v572: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:47.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.403+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28c80fde20 con 0x7f28c80f5510 2024-01-18T07:37:47.513 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.403+0000 7f28be7fc700 1 -- 172.21.15.84:0/4222435896 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f28c404de40 con 0x7f28c80f5510 2024-01-18T07:37:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:47.514 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 msgr2=0x7f28b405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 0x7f28b405e770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f28b80096c0 tx=0x7f28b8009be0).stop 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 msgr2=0x7f28c8106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f28c4004e30 tx=0x7f28c4009b30).stop 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 shutdown_connections 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f28c80f5510 0x7f28c8106ee0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f28b405c2b0 0x7f28b405e770 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 --2- 172.21.15.84:0/4222435896 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28c80f67b0 0x7f28c81073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 >> 172.21.15.84:0/4222435896 conn(0x7f28c80f0c70 msgr2=0x7f28c80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 shutdown_connections 2024-01-18T07:37:47.515 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:47.405+0000 7f28cea31700 1 -- 172.21.15.84:0/4222435896 wait complete. 2024-01-18T07:37:47.805 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:48.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:48 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:47.404427+0000 mon.smithi084 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.84:0/4222435896' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:48.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:48 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:47.404427+0000 mon.smithi084 (mon.0) 550 : audit [DBG] from='client.? 172.21.15.84:0/4222435896' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:49.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:49.044098+0000 mgr.smithi084.sihbqg (mgr.14162) 610 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:49.044098+0000 mgr.smithi084.sihbqg (mgr.14162) 610 : cluster [DBG] pgmap v573: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:50.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:51.040 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:51.376 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:51.044466+0000 mgr.smithi084.sihbqg (mgr.14162) 611 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:51.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:51.044466+0000 mgr.smithi084.sihbqg (mgr.14162) 611 : cluster [DBG] pgmap v574: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:52.053 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 -- 172.21.15.84:0/3249743743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 msgr2=0x7f25380f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 --2- 172.21.15.84:0/3249743743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25380f6bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f252c004660 tx=0x7f252c009b30).stop 2024-01-18T07:37:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 -- 172.21.15.84:0/3249743743 shutdown_connections 2024-01-18T07:37:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 --2- 172.21.15.84:0/3249743743 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 0x7f25380f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:37:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 --2- 172.21.15.84:0/3249743743 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25380f6bf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:52.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.048+0000 7f253f643700 1 -- 172.21.15.84:0/3249743743 >> 172.21.15.84:0/3249743743 conn(0x7f25380f1f50 msgr2=0x7f25380f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 -- 172.21.15.84:0/3249743743 shutdown_connections 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 -- 172.21.15.84:0/3249743743 wait complete. 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 Processor -- start 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 -- start start 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:52.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 0x7f25381075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538107c10 con 0x7f25380f67d0 2024-01-18T07:37:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253f643700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25381084c0 con 0x7f25380f7a70 2024-01-18T07:37:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253d3df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253d3df700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:54724/0 (socket says 172.21.15.84:54724) 2024-01-18T07:37:52.056 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253d3df700 1 -- 172.21.15.84:0/408146378 learned_addr learned my addr 172.21.15.84:0/408146378 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.049+0000 7f253cbde700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 0x7f25381075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253d3df700 1 -- 172.21.15.84:0/408146378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 msgr2=0x7f25381075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253d3df700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 0x7f25381075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253d3df700 1 -- 172.21.15.84:0/408146378 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c005040 con 0x7f25380f67d0 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253d3df700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f252c009f50 tx=0x7f252c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:52.057 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252c018070 con 0x7f25380f67d0 2024-01-18T07:37:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2538108740 con 0x7f25380f67d0 2024-01-18T07:37:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f252c013410 con 0x7f25380f67d0 2024-01-18T07:37:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252c01c620 con 0x7f25380f67d0 2024-01-18T07:37:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.050+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2538108ca0 con 0x7f25380f67d0 2024-01-18T07:37:52.058 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.051+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2538059100 con 0x7f25380f67d0 2024-01-18T07:37:52.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.051+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f252c01d420 con 0x7f25380f67d0 2024-01-18T07:37:52.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.052+0000 7f2529ffb700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 0x7f252405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:52.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.052+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f252c077a40 con 0x7f25380f67d0 2024-01-18T07:37:52.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.052+0000 7f253cbde700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 0x7f252405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:52.059 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.052+0000 7f253cbde700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 0x7f252405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f253400b790 tx=0x7f2534008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:52.060 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.056+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f252c04aa40 con 0x7f25380f67d0 2024-01-18T07:37:52.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.261+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2538027a00 con 0x7f25380f67d0 2024-01-18T07:37:52.262 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.261+0000 7f2529ffb700 1 -- 172.21.15.84:0/408146378 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f252c04e060 con 0x7f25380f67d0 2024-01-18T07:37:52.262 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:52.263 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 msgr2=0x7f252405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 0x7f252405e760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f253400b790 tx=0x7f2534008040).stop 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 msgr2=0x7f25381070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f252c009f50 tx=0x7f252c009b30).stop 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 shutdown_connections 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f25380f67d0 0x7f25381070e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f252405c2a0 0x7f252405e760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 --2- 172.21.15.84:0/408146378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f25380f7a70 0x7f25381075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 >> 172.21.15.84:0/408146378 conn(0x7f25380f1f50 msgr2=0x7f25380fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 shutdown_connections 2024-01-18T07:37:52.264 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:52.263+0000 7f253f643700 1 -- 172.21.15.84:0/408146378 wait complete. 2024-01-18T07:37:52.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:52 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:52.262628+0000 mon.smithi084 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.84:0/408146378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:52.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:52 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:52.262628+0000 mon.smithi084 (mon.0) 551 : audit [DBG] from='client.? 172.21.15.84:0/408146378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:52.675 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:53.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:53.044760+0000 mgr.smithi084.sihbqg (mgr.14162) 612 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:53.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:53.044760+0000 mgr.smithi084.sihbqg (mgr.14162) 612 : cluster [DBG] pgmap v575: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:55.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:55.045101+0000 mgr.smithi084.sihbqg (mgr.14162) 613 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:55.045101+0000 mgr.smithi084.sihbqg (mgr.14162) 613 : cluster [DBG] pgmap v576: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:55.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:37:55.911 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 -- 172.21.15.84:0/65414119 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 msgr2=0x7ff3200fbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 --2- 172.21.15.84:0/65414119 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3200fbd70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7ff310004660 tx=0x7ff310009b30).stop 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 -- 172.21.15.84:0/65414119 shutdown_connections 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 --2- 172.21.15.84:0/65414119 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3200fbd70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 --2- 172.21.15.84:0/65414119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 0x7ff3200f93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 -- 172.21.15.84:0/65414119 >> 172.21.15.84:0/65414119 conn(0x7ff3200f0e20 msgr2=0x7ff3200f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 -- 172.21.15.84:0/65414119 shutdown_connections 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.958+0000 7ff327729700 1 -- 172.21.15.84:0/65414119 wait complete. 2024-01-18T07:37:56.961 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 Processor -- start 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 -- start start 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 0x7ff320107100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320107ba0 con 0x7ff3200f9920 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff327729700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff320107ce0 con 0x7ff320060870 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff324cc4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff324cc4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:50990/0 (socket says 172.21.15.84:50990) 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff324cc4700 1 -- 172.21.15.84:0/2044165217 learned_addr learned my addr 172.21.15.84:0/2044165217 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.959+0000 7ff3254c5700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 0x7ff320107100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff324cc4700 1 -- 172.21.15.84:0/2044165217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 msgr2=0x7ff320107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff324cc4700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 0x7ff320107100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff324cc4700 1 -- 172.21.15.84:0/2044165217 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff310005040 con 0x7ff3200f9920 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff324cc4700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7ff310009830 tx=0x7ff310009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff310018070 con 0x7ff3200f9920 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff310007ba0 con 0x7ff3200f9920 2024-01-18T07:37:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff320108620 con 0x7ff3200f9920 2024-01-18T07:37:56.963 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff31001de20 con 0x7ff3200f9920 2024-01-18T07:37:56.963 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.960+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff320108b80 con 0x7ff3200f9920 2024-01-18T07:37:56.963 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.961+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff320045ad0 con 0x7ff3200f9920 2024-01-18T07:37:56.963 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.961+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7ff310007d10 con 0x7ff3200f9920 2024-01-18T07:37:56.964 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.962+0000 7ff315ffb700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 0x7ff30c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:37:56.964 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.962+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7ff310077710 con 0x7ff3200f9920 2024-01-18T07:37:56.964 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.962+0000 7ff3254c5700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 0x7ff30c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:37:56.964 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.962+0000 7ff3254c5700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 0x7ff30c05e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff31c004c40 tx=0x7ff31c00b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:37:56.966 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:56.966+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff31004a710 con 0x7ff3200f9920 2024-01-18T07:37:57.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.179+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff320027a00 con 0x7ff3200f9920 2024-01-18T07:37:57.180 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.179+0000 7ff315ffb700 1 -- 172.21.15.84:0/2044165217 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7ff31004dd30 con 0x7ff3200f9920 2024-01-18T07:37:57.180 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:37:57.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 msgr2=0x7ff30c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:57.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 0x7ff30c05e760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7ff31c004c40 tx=0x7ff31c00b040).stop 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 msgr2=0x7ff3201075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7ff310009830 tx=0x7ff310009b30).stop 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 shutdown_connections 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7ff30c05c2a0 0x7ff30c05e760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff320060870 0x7ff320107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 --2- 172.21.15.84:0/2044165217 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7ff3200f9920 0x7ff3201075f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 >> 172.21.15.84:0/2044165217 conn(0x7ff3200f0e20 msgr2=0x7ff3200f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 shutdown_connections 2024-01-18T07:37:57.182 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:37:57.181+0000 7ff327729700 1 -- 172.21.15.84:0/2044165217 wait complete. 2024-01-18T07:37:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:57.045422+0000 mgr.smithi084.sihbqg (mgr.14162) 614 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:57.045422+0000 mgr.smithi084.sihbqg (mgr.14162) 614 : cluster [DBG] pgmap v577: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:57.531 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:37:58.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:58 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:37:57.180463+0000 mon.smithi084 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.84:0/2044165217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:58.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:58 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:37:57.180463+0000 mon.smithi084 (mon.0) 552 : audit [DBG] from='client.? 172.21.15.84:0/2044165217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:37:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:37:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:37:59.045932+0000 mgr.smithi084.sihbqg (mgr.14162) 615 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:37:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:37:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:37:59.045932+0000 mgr.smithi084.sihbqg (mgr.14162) 615 : cluster [DBG] pgmap v578: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:00.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:00.766 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:01.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:01.046231+0000 mgr.smithi084.sihbqg (mgr.14162) 616 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:01.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:01.046231+0000 mgr.smithi084.sihbqg (mgr.14162) 616 : cluster [DBG] pgmap v579: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:01.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- 172.21.15.84:0/594512762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f56c0 msgr2=0x7f35880f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:01.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 --2- 172.21.15.84:0/594512762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f56c0 0x7f35880f5ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f357c004660 tx=0x7f357c009b30).stop 2024-01-18T07:38:01.713 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- 172.21.15.84:0/594512762 shutdown_connections 2024-01-18T07:38:01.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 --2- 172.21.15.84:0/594512762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f6960 0x7f35880f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:38:01.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 --2- 172.21.15.84:0/594512762 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f56c0 0x7f35880f5ae0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:01.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- 172.21.15.84:0/594512762 >> 172.21.15.84:0/594512762 conn(0x7f35880f0e20 msgr2=0x7f35880f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:01.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- 172.21.15.84:0/594512762 shutdown_connections 2024-01-18T07:38:01.714 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- 172.21.15.84:0/594512762 wait complete. 2024-01-18T07:38:01.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 Processor -- start 2024-01-18T07:38:01.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.713+0000 7f358d5e8700 1 -- start start 2024-01-18T07:38:01.715 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 0x7f35880f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880f8290 con 0x7f35880f6960 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880f83d0 con 0x7f35880f56c0 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:51014/0 (socket says 172.21.15.84:51014) 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 -- 172.21.15.84:0/3749483417 learned_addr learned my addr 172.21.15.84:0/3749483417 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f3586ffd700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 0x7f35880f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 -- 172.21.15.84:0/3749483417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 msgr2=0x7f35880f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 0x7f35880f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 -- 172.21.15.84:0/3749483417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c005040 con 0x7f35880f6960 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f3586ffd700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 0x7f35880f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f35867fc700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f357800b410 tx=0x7f3578006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:01.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35780125f0 con 0x7f35880f6960 2024-01-18T07:38:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35880f86b0 con 0x7f35880f6960 2024-01-18T07:38:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3578012c00 con 0x7f35880f6960 2024-01-18T07:38:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3578002aa0 con 0x7f35880f6960 2024-01-18T07:38:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.714+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358810f110 con 0x7f35880f6960 2024-01-18T07:38:01.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.715+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3588045ad0 con 0x7f35880f6960 2024-01-18T07:38:01.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.716+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f357800f030 con 0x7f35880f6960 2024-01-18T07:38:01.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.716+0000 7f356f7fe700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 0x7f357005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:01.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.716+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f3578076060 con 0x7f35880f6960 2024-01-18T07:38:01.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.716+0000 7f3586ffd700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 0x7f357005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:01.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.716+0000 7f3586ffd700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 0x7f357005e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f35880f95b0 tx=0x7f357c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:01.720 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.720+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3578049000 con 0x7f35880f6960 2024-01-18T07:38:01.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.929+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f35880fded0 con 0x7f35880f6960 2024-01-18T07:38:01.930 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.930+0000 7f356f7fe700 1 -- 172.21.15.84:0/3749483417 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f357804c620 con 0x7f35880f6960 2024-01-18T07:38:01.930 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 msgr2=0x7f357005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 0x7f357005e760 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f35880f95b0 tx=0x7f357c009b30).stop 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 msgr2=0x7f35880f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f357800b410 tx=0x7f3578006ef0).stop 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 shutdown_connections 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f357005c2a0 0x7f357005e760 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35880f56c0 0x7f35880f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 --2- 172.21.15.84:0/3749483417 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f35880f6960 0x7f35880f7ca0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:01.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.931+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 >> 172.21.15.84:0/3749483417 conn(0x7f35880f0e20 msgr2=0x7f35880603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:01.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.932+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 shutdown_connections 2024-01-18T07:38:01.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:01.932+0000 7f358d5e8700 1 -- 172.21.15.84:0/3749483417 wait complete. 2024-01-18T07:38:02.264 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:02 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:01.930916+0000 mon.smithi084 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.84:0/3749483417' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:02.314 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:02.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:02 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:01.930916+0000 mon.smithi084 (mon.0) 553 : audit [DBG] from='client.? 172.21.15.84:0/3749483417' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:03.046611+0000 mgr.smithi084.sihbqg (mgr.14162) 617 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:03.046611+0000 mgr.smithi084.sihbqg (mgr.14162) 617 : cluster [DBG] pgmap v580: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:05.316 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:05.047007+0000 mgr.smithi084.sihbqg (mgr.14162) 618 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:05.047007+0000 mgr.smithi084.sihbqg (mgr.14162) 618 : cluster [DBG] pgmap v581: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:05.552 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 -- 172.21.15.84:0/2236080366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f6650 msgr2=0x7f8ae40f6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/2236080366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f6650 0x7f8ae40f6ab0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f8ad8004660 tx=0x7f8ad8009b30).stop 2024-01-18T07:38:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 -- 172.21.15.84:0/2236080366 shutdown_connections 2024-01-18T07:38:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/2236080366 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f6650 0x7f8ae40f6ab0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.534 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/2236080366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae40f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 -- 172.21.15.84:0/2236080366 >> 172.21.15.84:0/2236080366 conn(0x7f8ae40f05e0 msgr2=0x7f8ae40f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 -- 172.21.15.84:0/2236080366 shutdown_connections 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 -- 172.21.15.84:0/2236080366 wait complete. 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.532+0000 7f8aea3d7700 1 Processor -- start 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8aea3d7700 1 -- start start 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8aea3d7700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8aea3d7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 0x7f8ae410c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8aea3d7700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40f5780 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8aea3d7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae40f58c0 con 0x7f8ae40f6650 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56352/0 (socket says 172.21.15.84:56352) 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 -- 172.21.15.84:0/783772772 learned_addr learned my addr 172.21.15.84:0/783772772 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae37fe700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 0x7f8ae410c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:06.535 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 -- 172.21.15.84:0/783772772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 msgr2=0x7f8ae410c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 0x7f8ae410c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 -- 172.21.15.84:0/783772772 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8005040 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae37fe700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 0x7f8ae410c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae3fff700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8ad400c530 tx=0x7f8ad4007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.533+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad400f650 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.534+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae410c870 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.534+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ad400fc60 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.534+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ad4012d20 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.536 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.534+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae410ce00 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.535+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8ad4019030 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.535+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae4059100 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.535+0000 7f8ae0ff9700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 0x7f8acc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.535+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8ad4075830 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.535+0000 7f8ae37fe700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 0x7f8acc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:06.537 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.536+0000 7f8ae37fe700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 0x7f8acc05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8ad8013580 tx=0x7f8ad8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:06.539 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.539+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ad4048880 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.747 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.747+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8ae4045ad0 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.748 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.747+0000 7f8ae0ff9700 1 -- 172.21.15.84:0/783772772 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8ae4045ad0 con 0x7f8ae40f5cb0 2024-01-18T07:38:06.748 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:06.749 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 msgr2=0x7f8acc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:06.749 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 0x7f8acc05e760 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8ad8013580 tx=0x7f8ad8009b30).stop 2024-01-18T07:38:06.749 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 msgr2=0x7f8ae4109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:06.749 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8ad400c530 tx=0x7f8ad4007e10).stop 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 shutdown_connections 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8ae40f5cb0 0x7f8ae4109dc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8acc05c2a0 0x7f8acc05e760 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 --2- 172.21.15.84:0/783772772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae40f6650 0x7f8ae410c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 >> 172.21.15.84:0/783772772 conn(0x7f8ae40f05e0 msgr2=0x7f8ae40fa550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 shutdown_connections 2024-01-18T07:38:06.750 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:06.749+0000 7f8aea3d7700 1 -- 172.21.15.84:0/783772772 wait complete. 2024-01-18T07:38:07.146 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:06.748337+0000 mon.smithi084 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.84:0/783772772' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:07.047315+0000 mgr.smithi084.sihbqg (mgr.14162) 619 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:06.748337+0000 mon.smithi084 (mon.0) 554 : audit [DBG] from='client.? 172.21.15.84:0/783772772' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:07.047315+0000 mgr.smithi084.sihbqg (mgr.14162) 619 : cluster [DBG] pgmap v582: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:09.047654+0000 mgr.smithi084.sihbqg (mgr.14162) 620 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:09.047654+0000 mgr.smithi084.sihbqg (mgr.14162) 620 : cluster [DBG] pgmap v583: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:10.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:10.378 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:11.225 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:11.047980+0000 mgr.smithi084.sihbqg (mgr.14162) 621 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- 172.21.15.84:0/2851120502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 msgr2=0x7fc1340f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 --2- 172.21.15.84:0/2851120502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc1340f5ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fc128004660 tx=0x7fc128009b30).stop 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- 172.21.15.84:0/2851120502 shutdown_connections 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 --2- 172.21.15.84:0/2851120502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc1340f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 --2- 172.21.15.84:0/2851120502 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc1340f5ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- 172.21.15.84:0/2851120502 >> 172.21.15.84:0/2851120502 conn(0x7fc1340f0e20 msgr2=0x7fc1340f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- 172.21.15.84:0/2851120502 shutdown_connections 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- 172.21.15.84:0/2851120502 wait complete. 2024-01-18T07:38:11.388 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 Processor -- start 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- start start 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc13410c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134060940 con 0x7fc1340f56c0 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.386+0000 7fc13b91d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc134060a80 con 0x7fc1340f6960 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56364/0 (socket says 172.21.15.84:56364) 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 -- 172.21.15.84:0/421969903 learned_addr learned my addr 172.21.15.84:0/421969903 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc138eb8700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc13410c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 -- 172.21.15.84:0/421969903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 msgr2=0x7fc13410c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc13410c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 -- 172.21.15.84:0/421969903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc128005040 con 0x7fc1340f56c0 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc138eb8700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc13410c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc1396b9700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fc128009f50 tx=0x7fc128009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:11.389 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc128018070 con 0x7fc1340f56c0 2024-01-18T07:38:11.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc128013410 con 0x7fc1340f56c0 2024-01-18T07:38:11.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13410c820 con 0x7fc1340f56c0 2024-01-18T07:38:11.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc12801c620 con 0x7fc1340f56c0 2024-01-18T07:38:11.390 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.387+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13410cd80 con 0x7fc1340f56c0 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.388+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc134045ad0 con 0x7fc1340f56c0 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.389+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fc12801d420 con 0x7fc1340f56c0 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.389+0000 7fc125ffb700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 0x7fc12005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.389+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fc128077970 con 0x7fc1340f56c0 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.389+0000 7fc138eb8700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 0x7fc12005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:11.391 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.389+0000 7fc138eb8700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 0x7fc12005e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc1300041d0 tx=0x7fc130009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:11.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.393+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc12804a9f0 con 0x7fc1340f56c0 2024-01-18T07:38:11.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:11.047980+0000 mgr.smithi084.sihbqg (mgr.14162) 621 : cluster [DBG] pgmap v584: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:11.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.599+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc1340fdf20 con 0x7fc1340f56c0 2024-01-18T07:38:11.600 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.600+0000 7fc125ffb700 1 -- 172.21.15.84:0/421969903 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fc12804e010 con 0x7fc1340f56c0 2024-01-18T07:38:11.600 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 msgr2=0x7fc12005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 0x7fc12005e760 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fc1300041d0 tx=0x7fc130009040).stop 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 msgr2=0x7fc134109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fc128009f50 tx=0x7fc128009b30).stop 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 shutdown_connections 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fc1340f56c0 0x7fc134109dd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fc12005c2a0 0x7fc12005e760 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 --2- 172.21.15.84:0/421969903 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1340f6960 0x7fc13410c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 >> 172.21.15.84:0/421969903 conn(0x7fc1340f0e20 msgr2=0x7fc1340fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:11.602 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.601+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 shutdown_connections 2024-01-18T07:38:11.603 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:11.602+0000 7fc13b91d700 1 -- 172.21.15.84:0/421969903 wait complete. 2024-01-18T07:38:11.995 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:12.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:12 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:11.600856+0000 mon.smithi084 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.84:0/421969903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:12.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:12 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:11.600856+0000 mon.smithi084 (mon.0) 555 : audit [DBG] from='client.? 172.21.15.84:0/421969903' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:13.048328+0000 mgr.smithi084.sihbqg (mgr.14162) 622 : cluster [DBG] pgmap v585: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:13.048328+0000 mgr.smithi084.sihbqg (mgr.14162) 622 : cluster [DBG] pgmap v585: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:14.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:15.235 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:15.339 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:15.048674+0000 mgr.smithi084.sihbqg (mgr.14162) 623 : cluster [DBG] pgmap v586: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:15.048674+0000 mgr.smithi084.sihbqg (mgr.14162) 623 : cluster [DBG] pgmap v586: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 -- 172.21.15.84:0/1429765506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f6960 msgr2=0x7f75f80f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/1429765506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f6960 0x7f75f80f6dc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f75e8004660 tx=0x7f75e8009b30).stop 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 -- 172.21.15.84:0/1429765506 shutdown_connections 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/1429765506 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f6960 0x7f75f80f6dc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/1429765506 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f56c0 0x7f75f80f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 -- 172.21.15.84:0/1429765506 >> 172.21.15.84:0/1429765506 conn(0x7f75f80f0e20 msgr2=0x7f75f80f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:16.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 -- 172.21.15.84:0/1429765506 shutdown_connections 2024-01-18T07:38:16.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.135+0000 7f75fd1b3700 1 -- 172.21.15.84:0/1429765506 wait complete. 2024-01-18T07:38:16.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 Processor -- start 2024-01-18T07:38:16.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 -- start start 2024-01-18T07:38:16.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:16.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f6960 0x7f75f81064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:16.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f8106a80 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75fd1b3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f8106bc0 con 0x7f75f80f6960 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75f6d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75f6d9d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:56388/0 (socket says 172.21.15.84:56388) 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75f6d9d700 1 -- 172.21.15.84:0/3564587320 learned_addr learned my addr 172.21.15.84:0/3564587320 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.136+0000 7f75f659c700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f6960 0x7f75f81064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75f6d9d700 1 -- 172.21.15.84:0/3564587320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f6960 msgr2=0x7f75f81064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75f6d9d700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f6960 0x7f75f81064d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75f6d9d700 1 -- 172.21.15.84:0/3564587320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e8005040 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75f6d9d700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f75ec002700 tx=0x7f75ec009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec0079c0 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75ec00f040 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75ec014860 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f8107560 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.137+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f8107b20 con 0x7f75f80f56c0 2024-01-18T07:38:16.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.138+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f8045ad0 con 0x7f75f80f56c0 2024-01-18T07:38:16.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.138+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f75ec01b060 con 0x7f75f80f56c0 2024-01-18T07:38:16.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.139+0000 7f75df7fe700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 0x7f75e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:16.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.139+0000 7f75f659c700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 0x7f75e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:16.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.139+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f75ec0759d0 con 0x7f75f80f56c0 2024-01-18T07:38:16.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.139+0000 7f75f659c700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 0x7f75e005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f75e8009f50 tx=0x7f75e8009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:16.143 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.143+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75ec0489d0 con 0x7f75f80f56c0 2024-01-18T07:38:16.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.348+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f75f80fdfd0 con 0x7f75f80f56c0 2024-01-18T07:38:16.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.348+0000 7f75df7fe700 1 -- 172.21.15.84:0/3564587320 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f75ec04bff0 con 0x7f75f80f56c0 2024-01-18T07:38:16.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 msgr2=0x7f75e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 0x7f75e005e760 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f75e8009f50 tx=0x7f75e8009b30).stop 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 msgr2=0x7f75f8105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f75ec002700 tx=0x7f75ec009d80).stop 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 shutdown_connections 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f75f80f56c0 0x7f75f8105fe0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f75e005c2a0 0x7f75e005e760 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 --2- 172.21.15.84:0/3564587320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f80f6960 0x7f75f81064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 >> 172.21.15.84:0/3564587320 conn(0x7f75f80f0e20 msgr2=0x7f75f80fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:16.351 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 shutdown_connections 2024-01-18T07:38:16.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:16.350+0000 7f75fd1b3700 1 -- 172.21.15.84:0/3564587320 wait complete. 2024-01-18T07:38:16.728 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:16.349401+0000 mon.smithi084 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.84:0/3564587320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:17.048987+0000 mgr.smithi084.sihbqg (mgr.14162) 624 : cluster [DBG] pgmap v587: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:16.349401+0000 mon.smithi084 (mon.0) 556 : audit [DBG] from='client.? 172.21.15.84:0/3564587320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:17.048987+0000 mgr.smithi084.sihbqg (mgr.14162) 624 : cluster [DBG] pgmap v587: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:18.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:38:17.702806+0000 mgr.smithi084.sihbqg (mgr.14162) 625 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:38:18.383 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: yield (conn, connr) 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: code, '\n'.join(err))) 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:38:18.384 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: Traceback (most recent call last): 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: main() 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: r = ctx.func(ctx) 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: return func(ctx) 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: ports=daemon_ports) 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.385 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:18 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:17.703441+0000 mon.smithi084 (mon.0) 557 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:38:17.702806+0000 mgr.smithi084.sihbqg (mgr.14162) 625 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:38:18.415 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: yield (conn, connr) 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: code, '\n'.join(err))) 2024-01-18T07:38:18.416 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Reconfig daemon node-exporter.smithi106 ... 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Non-zero exit code 1 from systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr Job for ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service failed because a timeout was exceeded. 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: systemctl: stderr See "systemctl status ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106.service" and "journalctl -xe" for details. 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: Traceback (most recent call last): 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: main() 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: r = ctx.func(ctx) 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: return func(ctx) 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: ports=daemon_ports) 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2637, in deploy_daemon 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: get_unit_name(fsid, daemon_type, daemon_id)]) 2024-01-18T07:38:18.417 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: File "/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-18T07:38:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-18T07:38:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: RuntimeError: Failed command: systemctl restart ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@node-exporter.smithi106 2024-01-18T07:38:18.418 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:18 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:17.703441+0000 mon.smithi084 (mon.0) 557 : audit [DBG] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-18T07:38:19.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:19.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:18.614766+0000 mon.smithi084 (mon.0) 558 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:19.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:19.049295+0000 mgr.smithi084.sihbqg (mgr.14162) 626 : cluster [DBG] pgmap v588: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:19.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:18.614766+0000 mon.smithi084 (mon.0) 558 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:19.915 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:19.049295+0000 mgr.smithi084.sihbqg (mgr.14162) 626 : cluster [DBG] pgmap v588: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:21.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- 172.21.15.84:0/1112658322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f6960 msgr2=0x7fab500f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:21.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 --2- 172.21.15.84:0/1112658322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f6960 0x7fab500f6dc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fab4c004660 tx=0x7fab4c009b30).stop 2024-01-18T07:38:21.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- 172.21.15.84:0/1112658322 shutdown_connections 2024-01-18T07:38:21.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 --2- 172.21.15.84:0/1112658322 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f6960 0x7fab500f6dc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.014 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 --2- 172.21.15.84:0/1112658322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f56c0 0x7fab500f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- 172.21.15.84:0/1112658322 >> 172.21.15.84:0/1112658322 conn(0x7fab500f0e20 msgr2=0x7fab500f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- 172.21.15.84:0/1112658322 shutdown_connections 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- 172.21.15.84:0/1112658322 wait complete. 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 Processor -- start 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.012+0000 7fab574a6700 1 -- start start 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab574a6700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab574a6700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f6960 0x7fab5010c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab574a6700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab50060920 con 0x7fab500f56c0 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab574a6700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab50060a60 con 0x7fab500f6960 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53436/0 (socket says 172.21.15.84:53436) 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 -- 172.21.15.84:0/2431358883 learned_addr learned my addr 172.21.15.84:0/2431358883 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab54a41700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f6960 0x7fab5010c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 -- 172.21.15.84:0/2431358883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f6960 msgr2=0x7fab5010c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:21.015 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f6960 0x7fab5010c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 -- 172.21.15.84:0/2431358883 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c005040 con 0x7fab500f56c0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab55242700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fab40002700 tx=0x7fab40009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab400079c0 con 0x7fab500f56c0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5010c8b0 con 0x7fab500f56c0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab4000f040 con 0x7fab500f56c0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.013+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab40014860 con 0x7fab500f56c0 2024-01-18T07:38:21.016 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.014+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5010ce40 con 0x7fab500f56c0 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.014+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab50059100 con 0x7fab500f56c0 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.015+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fab4001b060 con 0x7fab500f56c0 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.015+0000 7fab45ffb700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 0x7fab3c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.015+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fab40075a00 con 0x7fab500f56c0 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.015+0000 7fab54a41700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 0x7fab3c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:21.017 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.015+0000 7fab54a41700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 0x7fab3c05e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fab4c009f50 tx=0x7fab4c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:21.020 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.019+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab40048a00 con 0x7fab500f56c0 2024-01-18T07:38:21.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.241+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fab50045ad0 con 0x7fab500f56c0 2024-01-18T07:38:21.242 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.241+0000 7fab45ffb700 1 -- 172.21.15.84:0/2431358883 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fab4004c020 con 0x7fab500f56c0 2024-01-18T07:38:21.243 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:21.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 msgr2=0x7fab3c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 0x7fab3c05e760 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fab4c009f50 tx=0x7fab4c009b30).stop 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 msgr2=0x7fab50109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fab40002700 tx=0x7fab40009d80).stop 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 shutdown_connections 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fab500f56c0 0x7fab50109e00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.245 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fab3c05c2a0 0x7fab3c05e760 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 --2- 172.21.15.84:0/2431358883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fab500f6960 0x7fab5010c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:21.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.243+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 >> 172.21.15.84:0/2431358883 conn(0x7fab500f0e20 msgr2=0x7fab500fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:21.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.244+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 shutdown_connections 2024-01-18T07:38:21.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:21.244+0000 7fab574a6700 1 -- 172.21.15.84:0/2431358883 wait complete. 2024-01-18T07:38:21.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:20.178018+0000 mon.smithi084 (mon.0) 559 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:21.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:20.946322+0000 mon.smithi084 (mon.0) 560 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:21.465 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:21.049606+0000 mgr.smithi084.sihbqg (mgr.14162) 627 : cluster [DBG] pgmap v589: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:21.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:20.178018+0000 mon.smithi084 (mon.0) 559 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:21.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:20.946322+0000 mon.smithi084 (mon.0) 560 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:21.665 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:21.049606+0000 mgr.smithi084.sihbqg (mgr.14162) 627 : cluster [DBG] pgmap v589: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:21.702 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:22.544 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:22 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:21.242615+0000 mon.smithi084 (mon.0) 561 : audit [DBG] from='client.? 172.21.15.84:0/2431358883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:22.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:22 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:21.242615+0000 mon.smithi084 (mon.0) 561 : audit [DBG] from='client.? 172.21.15.84:0/2431358883' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:23.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:23.049921+0000 mgr.smithi084.sihbqg (mgr.14162) 628 : cluster [DBG] pgmap v590: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:23.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:23.080948+0000 mon.smithi084 (mon.0) 562 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:23.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:23.049921+0000 mgr.smithi084.sihbqg (mgr.14162) 628 : cluster [DBG] pgmap v590: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:23.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:23.080948+0000 mon.smithi084 (mon.0) 562 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:24.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:23.314271+0000 mon.smithi084 (mon.0) 563 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:24.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:24 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cephadm 2024-01-18T07:38:23.317980+0000 mgr.smithi084.sihbqg (mgr.14162) 629 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:38:24.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:23.314271+0000 mon.smithi084 (mon.0) 563 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' 2024-01-18T07:38:24.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:24 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cephadm 2024-01-18T07:38:23.317980+0000 mgr.smithi084.sihbqg (mgr.14162) 629 : cephadm [INF] Deploying daemon prometheus.smithi106 on smithi106 2024-01-18T07:38:24.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:24.935 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:25.608 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:25.050204+0000 mgr.smithi084.sihbqg (mgr.14162) 630 : cluster [DBG] pgmap v591: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:25.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:25.050204+0000 mgr.smithi084.sihbqg (mgr.14162) 630 : cluster [DBG] pgmap v591: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 -- 172.21.15.84:0/1530367660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f5280 msgr2=0x7f300c0f7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 --2- 172.21.15.84:0/1530367660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f5280 0x7f300c0f7a00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f3000004660 tx=0x7f3000009b30).stop 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 -- 172.21.15.84:0/1530367660 shutdown_connections 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 --2- 172.21.15.84:0/1530367660 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f5280 0x7f300c0f7a00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 --2- 172.21.15.84:0/1530367660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f48e0 0x7f300c0f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 -- 172.21.15.84:0/1530367660 >> 172.21.15.84:0/1530367660 conn(0x7f300c0f0400 msgr2=0x7f300c0f2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 -- 172.21.15.84:0/1530367660 shutdown_connections 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.919+0000 7f301160a700 1 -- 172.21.15.84:0/1530367660 wait complete. 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 Processor -- start 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 -- start start 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:25.922 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f5280 0x7f300c0f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0f5f00 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f301160a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0f6040 con 0x7f300c0f5280 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f300bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f300bfff700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:53454/0 (socket says 172.21.15.84:53454) 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.920+0000 7f300bfff700 1 -- 172.21.15.84:0/3277257509 learned_addr learned my addr 172.21.15.84:0/3277257509 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f300b7fe700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f5280 0x7f300c0f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f300bfff700 1 -- 172.21.15.84:0/3277257509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f5280 msgr2=0x7f300c0f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f300bfff700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f5280 0x7f300c0f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f300bfff700 1 -- 172.21.15.84:0/3277257509 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000005040 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f300bfff700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f2ffc0093f0 tx=0x7f2ffc006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc013640 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ffc013c50 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c0f6320 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ffc012c80 con 0x7f300c0f48e0 2024-01-18T07:38:25.923 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.921+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c10ef80 con 0x7f300c0f48e0 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.922+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c045ad0 con 0x7f300c0f48e0 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.922+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f2ffc01a060 con 0x7f300c0f48e0 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.923+0000 7f3008ff9700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 0x7f2ff405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.923+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f2ffc075b90 con 0x7f300c0f48e0 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.923+0000 7f300b7fe700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 0x7f2ff405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:25.924 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.923+0000 7f300b7fe700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 0x7f2ff405e770 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f3000004630 tx=0x7f3000009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:25.927 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:25.926+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ffc048b90 con 0x7f300c0f48e0 2024-01-18T07:38:26.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.136+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f300c027a00 con 0x7f300c0f48e0 2024-01-18T07:38:26.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.136+0000 7f3008ff9700 1 -- 172.21.15.84:0/3277257509 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f2ffc04c1b0 con 0x7f300c0f48e0 2024-01-18T07:38:26.136 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 msgr2=0x7f2ff405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 0x7f2ff405e770 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f3000004630 tx=0x7f3000009b30).stop 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 msgr2=0x7f300c0f7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f2ffc0093f0 tx=0x7f2ffc006ec0).stop 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 shutdown_connections 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f300c0f48e0 0x7f300c0f7490 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:26.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f2ff405c2b0 0x7f2ff405e770 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:26.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 --2- 172.21.15.84:0/3277257509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f300c0f5280 0x7f300c0f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:26.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 >> 172.21.15.84:0/3277257509 conn(0x7f300c0f0400 msgr2=0x7f300c0fb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:26.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 shutdown_connections 2024-01-18T07:38:26.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:26.138+0000 7f301160a700 1 -- 172.21.15.84:0/3277257509 wait complete. 2024-01-18T07:38:26.518 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:26.137100+0000 mon.smithi084 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.84:0/3277257509' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:27.050647+0000 mgr.smithi084.sihbqg (mgr.14162) 631 : cluster [DBG] pgmap v592: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:26.137100+0000 mon.smithi084 (mon.0) 564 : audit [DBG] from='client.? 172.21.15.84:0/3277257509' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:27.050647+0000 mgr.smithi084.sihbqg (mgr.14162) 631 : cluster [DBG] pgmap v592: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:29.050965+0000 mgr.smithi084.sihbqg (mgr.14162) 632 : cluster [DBG] pgmap v593: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:29.050965+0000 mgr.smithi084.sihbqg (mgr.14162) 632 : cluster [DBG] pgmap v593: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:29.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:29.754 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- 172.21.15.84:0/4279321010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 msgr2=0x7f8cdc0f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/4279321010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc0f4ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f8cd40099e0 tx=0x7f8cd4004d10).stop 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- 172.21.15.84:0/4279321010 shutdown_connections 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/4279321010 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc0f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/4279321010 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc0f4ec0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- 172.21.15.84:0/4279321010 >> 172.21.15.84:0/4279321010 conn(0x7f8cdc0f05e0 msgr2=0x7f8cdc0f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- 172.21.15.84:0/4279321010 shutdown_connections 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- 172.21.15.84:0/4279321010 wait complete. 2024-01-18T07:38:30.767 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 Processor -- start 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.764+0000 7f8ce30c3700 1 -- start start 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc107700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc107cb0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc107df0 con 0x7f8cdc0f5440 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:38506/0 (socket says 172.21.15.84:38506) 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 -- 172.21.15.84:0/960655025 learned_addr learned my addr 172.21.15.84:0/960655025 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8cdbfff700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc107700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 -- 172.21.15.84:0/960655025 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 msgr2=0x7f8cdc107700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc107700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 -- 172.21.15.84:0/960655025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4009640 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8cdbfff700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce0e5f700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f8cd4007830 tx=0x7f8cd4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:30.768 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cd400e030 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cd401d8d0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc108730 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cd4018070 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.769 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.765+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc108c10 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.766+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc045ad0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.767+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f8cd401da40 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.767+0000 7f8cd97fa700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 0x7f8cc805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.767+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f8cd4077ad0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.767+0000 7f8cdbfff700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 0x7f8cc805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:30.770 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.767+0000 7f8cdbfff700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 0x7f8cc805e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8cdc0f5880 tx=0x7f8cc4009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:30.772 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.771+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cd404aad0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.978+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8cdc0fefb0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.980 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.979+0000 7f8cd97fa700 1 -- 172.21.15.84:0/960655025 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f8cd404e0f0 con 0x7f8cdc0f4aa0 2024-01-18T07:38:30.980 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 msgr2=0x7f8cc805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 0x7f8cc805e760 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8cdc0f5880 tx=0x7f8cc4009040).stop 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 msgr2=0x7f8cdc107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f8cd4007830 tx=0x7f8cd4004d10).stop 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 shutdown_connections 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f8cdc0f4aa0 0x7f8cdc107210 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f8cc805c2a0 0x7f8cc805e760 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.982 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 --2- 172.21.15.84:0/960655025 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8cdc0f5440 0x7f8cdc107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:30.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.980+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 >> 172.21.15.84:0/960655025 conn(0x7f8cdc0f05e0 msgr2=0x7f8cdc0fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:30.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.981+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 shutdown_connections 2024-01-18T07:38:30.983 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:30.981+0000 7f8ce30c3700 1 -- 172.21.15.84:0/960655025 wait complete. 2024-01-18T07:38:31.324 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:30.980934+0000 mon.smithi084 (mon.0) 565 : audit [DBG] from='client.? 172.21.15.84:0/960655025' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:31.391 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:30.980934+0000 mon.smithi084 (mon.0) 565 : audit [DBG] from='client.? 172.21.15.84:0/960655025' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:32.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:32 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:31.051270+0000 mgr.smithi084.sihbqg (mgr.14162) 633 : cluster [DBG] pgmap v594: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:32.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:32 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:31.051270+0000 mgr.smithi084.sihbqg (mgr.14162) 633 : cluster [DBG] pgmap v594: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:33.051730+0000 mgr.smithi084.sihbqg (mgr.14162) 634 : cluster [DBG] pgmap v595: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:33.051730+0000 mgr.smithi084.sihbqg (mgr.14162) 634 : cluster [DBG] pgmap v595: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:34.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:34.628 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:35.051960+0000 mgr.smithi084.sihbqg (mgr.14162) 635 : cluster [DBG] pgmap v596: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:35.065601+0000 mon.smithi084 (mon.0) 566 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:38:35.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:35.068406+0000 mon.smithi084 (mon.0) 567 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:38:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:35.051960+0000 mgr.smithi084.sihbqg (mgr.14162) 635 : cluster [DBG] pgmap v596: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:35.065601+0000 mon.smithi084 (mon.0) 566 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/mirror_snapshot_schedule"}]: dispatch 2024-01-18T07:38:35.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:35 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:35.068406+0000 mon.smithi084 (mon.0) 567 : audit [INF] from='mgr.14162 172.21.15.84:0/3357745669' entity='mgr.smithi084.sihbqg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi084.sihbqg/trash_purge_schedule"}]: dispatch 2024-01-18T07:38:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 -- 172.21.15.84:0/1438939068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f56c0 msgr2=0x7f637c0f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 --2- 172.21.15.84:0/1438939068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f56c0 0x7f637c0f5ae0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6364004660 tx=0x7f6364009b30).stop 2024-01-18T07:38:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 -- 172.21.15.84:0/1438939068 shutdown_connections 2024-01-18T07:38:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 --2- 172.21.15.84:0/1438939068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f6960 0x7f637c0f6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.591 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 --2- 172.21.15.84:0/1438939068 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f56c0 0x7f637c0f5ae0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 -- 172.21.15.84:0/1438939068 >> 172.21.15.84:0/1438939068 conn(0x7f637c0f0e20 msgr2=0x7f637c0f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 -- 172.21.15.84:0/1438939068 shutdown_connections 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.588+0000 7f6380c29700 1 -- 172.21.15.84:0/1438939068 wait complete. 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 Processor -- start 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 -- start start 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f56c0 0x7f637c107050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c107af0 con 0x7f637c0f6960 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6380c29700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c107c30 con 0x7f637c0f56c0 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:38516/0 (socket says 172.21.15.84:38516) 2024-01-18T07:38:35.592 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 -- 172.21.15.84:0/1778857054 learned_addr learned my addr 172.21.15.84:0/1778857054 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f637a59c700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f56c0 0x7f637c107050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 -- 172.21.15.84:0/1778857054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f56c0 msgr2=0x7f637c107050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f56c0 0x7f637c107050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 -- 172.21.15.84:0/1778857054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364005040 con 0x7f637c0f6960 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6379d9b700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f636c00e7f0 tx=0x7f636c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.589+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636c0092e0 con 0x7f637c0f6960 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.590+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f636c00f040 con 0x7f637c0f6960 2024-01-18T07:38:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.590+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c1085d0 con 0x7f637c0f6960 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.590+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f636c014750 con 0x7f637c0f6960 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.590+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c108b90 con 0x7f637c0f6960 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.590+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c045ad0 con 0x7f637c0f6960 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.591+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f636c01b030 con 0x7f637c0f6960 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.591+0000 7f6372ffd700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 0x7f636805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:35.594 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.592+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f636c0759a0 con 0x7f637c0f6960 2024-01-18T07:38:35.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.592+0000 7f637a59c700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 0x7f636805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:35.595 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.592+0000 7f637a59c700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 0x7f636805e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6364013530 tx=0x7f6364016040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:35.596 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.595+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f636c048a70 con 0x7f637c0f6960 2024-01-18T07:38:35.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.802+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f637c109220 con 0x7f637c0f6960 2024-01-18T07:38:35.803 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.802+0000 7f6372ffd700 1 -- 172.21.15.84:0/1778857054 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f636c019020 con 0x7f637c0f6960 2024-01-18T07:38:35.804 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 msgr2=0x7f636805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 0x7f636805e760 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6364013530 tx=0x7f6364016040).stop 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 msgr2=0x7f637c107540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f636c00e7f0 tx=0x7f636c00b040).stop 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 shutdown_connections 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f636805c2a0 0x7f636805e760 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f637c0f56c0 0x7f637c107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 --2- 172.21.15.84:0/1778857054 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f637c0f6960 0x7f637c107540 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:35.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 >> 172.21.15.84:0/1778857054 conn(0x7f637c0f0e20 msgr2=0x7f637c0603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:35.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 shutdown_connections 2024-01-18T07:38:35.806 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:35.804+0000 7f6380c29700 1 -- 172.21.15.84:0/1778857054 wait complete. 2024-01-18T07:38:36.182 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:36.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:35.804146+0000 mon.smithi084 (mon.0) 568 : audit [DBG] from='client.? 172.21.15.84:0/1778857054' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:35.804146+0000 mon.smithi084 (mon.0) 568 : audit [DBG] from='client.? 172.21.15.84:0/1778857054' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:37.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:37 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:37.052395+0000 mgr.smithi084.sihbqg (mgr.14162) 636 : cluster [DBG] pgmap v597: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:37.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:37 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:37.052395+0000 mgr.smithi084.sihbqg (mgr.14162) 636 : cluster [DBG] pgmap v597: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:39.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:39.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:39.052898+0000 mgr.smithi084.sihbqg (mgr.14162) 637 : cluster [DBG] pgmap v598: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:39.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:39 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:39.052898+0000 mgr.smithi084.sihbqg (mgr.14162) 637 : cluster [DBG] pgmap v598: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:39.417 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3470280630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f5440 msgr2=0x7f33f80f7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3470280630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f5440 0x7f33f80f7bc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f33ec004660 tx=0x7f33ec009b30).stop 2024-01-18T07:38:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3470280630 shutdown_connections 2024-01-18T07:38:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3470280630 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f5440 0x7f33f80f7bc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.392 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3470280630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f4aa0 0x7f33f80f4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3470280630 >> 172.21.15.84:0/3470280630 conn(0x7f33f80f05e0 msgr2=0x7f33f80f2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:40.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3470280630 shutdown_connections 2024-01-18T07:38:40.393 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.390+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3470280630 wait complete. 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.391+0000 7f33fd3ae700 1 Processor -- start 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.391+0000 7f33fd3ae700 1 -- start start 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.391+0000 7f33fd3ae700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.391+0000 7f33fd3ae700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 0x7f33f81075c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.391+0000 7f33fd3ae700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8107b70 con 0x7f33f80f4aa0 2024-01-18T07:38:40.395 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33fd3ae700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f8107cb0 con 0x7f33f80f5440 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42728/0 (socket says 172.21.15.84:42728) 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 -- 172.21.15.84:0/3255395390 learned_addr learned my addr 172.21.15.84:0/3255395390 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f67fc700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 0x7f33f81075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 -- 172.21.15.84:0/3255395390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 msgr2=0x7f33f81075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 0x7f33f81075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 -- 172.21.15.84:0/3255395390 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec005040 con 0x7f33f80f4aa0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f67fc700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 0x7f33f81075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33f6ffd700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f33e8002700 tx=0x7f33e8009d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e8009320 con 0x7f33f80f4aa0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33e800f040 con 0x7f33f80f4aa0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f8108650 con 0x7f33f80f4aa0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e8014900 con 0x7f33f80f4aa0 2024-01-18T07:38:40.396 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.392+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f8108c10 con 0x7f33f80f4aa0 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.393+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f8045ad0 con 0x7f33f80f4aa0 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.393+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f33e8009480 con 0x7f33f80f4aa0 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.394+0000 7f33df7fe700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 0x7f33e005e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.394+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f33e8075ed0 con 0x7f33f80f4aa0 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.394+0000 7f33f67fc700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 0x7f33e005e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:40.397 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.394+0000 7f33f67fc700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 0x7f33e005e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f33ec009d60 tx=0x7f33ec009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:40.399 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.398+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33e8048ed0 con 0x7f33f80f4aa0 2024-01-18T07:38:40.604 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.603+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f33f80ff110 con 0x7f33f80f4aa0 2024-01-18T07:38:40.605 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.603+0000 7f33df7fe700 1 -- 172.21.15.84:0/3255395390 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f33e804c4f0 con 0x7f33f80f4aa0 2024-01-18T07:38:40.605 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:40.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 msgr2=0x7f33e005e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:40.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 0x7f33e005e760 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f33ec009d60 tx=0x7f33ec009b30).stop 2024-01-18T07:38:40.606 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 msgr2=0x7f33f81070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f33e8002700 tx=0x7f33e8009d80).stop 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 shutdown_connections 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f33f80f4aa0 0x7f33f81070d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f33e005c2a0 0x7f33e005e760 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 --2- 172.21.15.84:0/3255395390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33f80f5440 0x7f33f81075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 >> 172.21.15.84:0/3255395390 conn(0x7f33f80f05e0 msgr2=0x7f33f80fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 shutdown_connections 2024-01-18T07:38:40.607 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:40.605+0000 7f33fd3ae700 1 -- 172.21.15.84:0/3255395390 wait complete. 2024-01-18T07:38:40.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:40 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:40.605294+0000 mon.smithi084 (mon.0) 569 : audit [DBG] from='client.? 172.21.15.84:0/3255395390' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:40.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:40.605294+0000 mon.smithi084 (mon.0) 569 : audit [DBG] from='client.? 172.21.15.84:0/3255395390' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:40.983 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:41.914 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:41 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:41.053255+0000 mgr.smithi084.sihbqg (mgr.14162) 638 : cluster [DBG] pgmap v599: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:42.132 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:41 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:41.053255+0000 mgr.smithi084.sihbqg (mgr.14162) 638 : cluster [DBG] pgmap v599: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:43.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:43 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:43.053748+0000 mgr.smithi084.sihbqg (mgr.14162) 639 : cluster [DBG] pgmap v600: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:43.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:43.053748+0000 mgr.smithi084.sihbqg (mgr.14162) 639 : cluster [DBG] pgmap v600: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:43.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:44.219 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:45.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.242+0000 7f4a04ccb700 1 -- 172.21.15.84:0/396324999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 msgr2=0x7f4a000f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:45.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.242+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/396324999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a000f7d10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f49f4004660 tx=0x7f49f4009b30).stop 2024-01-18T07:38:45.243 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- 172.21.15.84:0/396324999 shutdown_connections 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/396324999 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a000f7d10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/396324999 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a000f6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- 172.21.15.84:0/396324999 >> 172.21.15.84:0/396324999 conn(0x7f4a000f1d70 msgr2=0x7f4a000f4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- 172.21.15.84:0/396324999 shutdown_connections 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- 172.21.15.84:0/396324999 wait complete. 2024-01-18T07:38:45.244 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 Processor -- start 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- start start 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a00106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a00107970 con 0x7f4a000f78b0 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.243+0000 7f4a04ccb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a00107ab0 con 0x7f4a000f6610 2024-01-18T07:38:45.246 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42760/0 (socket says 172.21.15.84:42760) 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 -- 172.21.15.84:0/807173403 learned_addr learned my addr 172.21.15.84:0/807173403 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49ff7fe700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a00106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 -- 172.21.15.84:0/807173403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 msgr2=0x7f4a00106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a00106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 -- 172.21.15.84:0/807173403 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4005040 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49ff7fe700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a00106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49feffd700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f49f4009830 tx=0x7f49f4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49f4018070 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f49f4013410 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a001083f0 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49f401c620 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.244+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a00108950 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.245+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a00045ad0 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.245+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f49f401c890 con 0x7f4a000f78b0 2024-01-18T07:38:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.246+0000 7f49e7fff700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 0x7f49e805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:45.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.246+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f49f40779b0 con 0x7f4a000f78b0 2024-01-18T07:38:45.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.246+0000 7f49ff7fe700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 0x7f49e805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:45.248 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.246+0000 7f49ff7fe700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 0x7f49e805e770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4a000f7710 tx=0x7f49f8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:45.250 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.250+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49f404a9d0 con 0x7f4a000f78b0 2024-01-18T07:38:45.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:45 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:45.054266+0000 mgr.smithi084.sihbqg (mgr.14162) 640 : cluster [DBG] pgmap v601: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:45.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:45 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:45.054266+0000 mgr.smithi084.sihbqg (mgr.14162) 640 : cluster [DBG] pgmap v601: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:45.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.459+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4a00027a00 con 0x7f4a000f78b0 2024-01-18T07:38:45.460 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.459+0000 7f49e7fff700 1 -- 172.21.15.84:0/807173403 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f49f404dff0 con 0x7f4a000f78b0 2024-01-18T07:38:45.460 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 msgr2=0x7f49e805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 0x7f49e805e770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4a000f7710 tx=0x7f49f8009040).stop 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 msgr2=0x7f4a001073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f49f4009830 tx=0x7f49f4009b30).stop 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 shutdown_connections 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f49e805c2b0 0x7f49e805e770 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a000f6610 0x7f4a00106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 --2- 172.21.15.84:0/807173403 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4a000f78b0 0x7f4a001073c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 >> 172.21.15.84:0/807173403 conn(0x7f4a000f1d70 msgr2=0x7f4a000fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 shutdown_connections 2024-01-18T07:38:45.462 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:45.461+0000 7f4a04ccb700 1 -- 172.21.15.84:0/807173403 wait complete. 2024-01-18T07:38:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:46.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:46 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:45.460510+0000 mon.smithi084 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.84:0/807173403' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:46.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:46 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:45.460510+0000 mon.smithi084 (mon.0) 570 : audit [DBG] from='client.? 172.21.15.84:0/807173403' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:47.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:47 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:47.054684+0000 mgr.smithi084.sihbqg (mgr.14162) 641 : cluster [DBG] pgmap v602: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:47.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:47 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:47.054684+0000 mgr.smithi084.sihbqg (mgr.14162) 641 : cluster [DBG] pgmap v602: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:48.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:49.105 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:49.243 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:49 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:49.055024+0000 mgr.smithi084.sihbqg (mgr.14162) 642 : cluster [DBG] pgmap v603: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:49.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:49 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:49.055024+0000 mgr.smithi084.sihbqg (mgr.14162) 642 : cluster [DBG] pgmap v603: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:50.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 -- 172.21.15.84:0/666816165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f6980 msgr2=0x7f4f7c0f6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:50.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 --2- 172.21.15.84:0/666816165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c0f6de0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4f68004660 tx=0x7f4f68009b30).stop 2024-01-18T07:38:50.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 -- 172.21.15.84:0/666816165 shutdown_connections 2024-01-18T07:38:50.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 --2- 172.21.15.84:0/666816165 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c0f6de0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.137 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 --2- 172.21.15.84:0/666816165 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c0f5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 -- 172.21.15.84:0/666816165 >> 172.21.15.84:0/666816165 conn(0x7f4f7c0f0e20 msgr2=0x7f4f7c0f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 -- 172.21.15.84:0/666816165 shutdown_connections 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.135+0000 7f4f82c77700 1 -- 172.21.15.84:0/666816165 wait complete. 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 Processor -- start 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 -- start start 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c106a40 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f82c77700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7c106b80 con 0x7f4f7c0f6980 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35824/0 (socket says 172.21.15.84:35824) 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 -- 172.21.15.84:0/4006348946 learned_addr learned my addr 172.21.15.84:0/4006348946 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f7bfff700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:50.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 -- 172.21.15.84:0/4006348946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f6980 msgr2=0x7f4f7c106490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c106490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.136+0000 7f4f80a13700 1 -- 172.21.15.84:0/4006348946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f68005040 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f80a13700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4f7000e9a0 tx=0x7f4f70007ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f7000c9b0 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c107520 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f70011470 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f70010630 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.139 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.137+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7c107ae0 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.138+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c045ad0 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.138+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4f70021070 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.138+0000 7f4f797fa700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.138+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4f70078760 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.138+0000 7f4f7bfff700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:50.140 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.139+0000 7f4f7bfff700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f4f68013580 tx=0x7f4f68009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:50.142 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.142+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f7004b760 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.349+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4f7c108180 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.350 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.349+0000 7f4f797fa700 1 -- 172.21.15.84:0/4006348946 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f4f70015080 con 0x7f4f7c0f56e0 2024-01-18T07:38:50.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 msgr2=0x7f4f6c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f4f68013580 tx=0x7f4f68009b30).stop 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 msgr2=0x7f4f7c105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4f7000e9a0 tx=0x7f4f70007ea0).stop 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 shutdown_connections 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4f7c0f56e0 0x7f4f7c105fa0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4f6c05c2a0 0x7f4f6c05e760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 --2- 172.21.15.84:0/4006348946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f7c0f6980 0x7f4f7c106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 >> 172.21.15.84:0/4006348946 conn(0x7f4f7c0f0e20 msgr2=0x7f4f7c0fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 shutdown_connections 2024-01-18T07:38:50.352 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:50.351+0000 7f4f82c77700 1 -- 172.21.15.84:0/4006348946 wait complete. 2024-01-18T07:38:50.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:50 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:50.350560+0000 mon.smithi084 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.84:0/4006348946' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:50.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:50 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:50.350560+0000 mon.smithi084 (mon.0) 571 : audit [DBG] from='client.? 172.21.15.84:0/4006348946' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:50.774 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:51.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:51 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:51.055481+0000 mgr.smithi084.sihbqg (mgr.14162) 643 : cluster [DBG] pgmap v604: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:51.882 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:51 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:51.055481+0000 mgr.smithi084.sihbqg (mgr.14162) 643 : cluster [DBG] pgmap v604: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:53.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:53 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:53.056035+0000 mgr.smithi084.sihbqg (mgr.14162) 644 : cluster [DBG] pgmap v605: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:53.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:53 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:53.056035+0000 mgr.smithi084.sihbqg (mgr.14162) 644 : cluster [DBG] pgmap v605: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:53.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:54.009 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 -- 172.21.15.84:0/3762387444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f5510 msgr2=0x7f38300f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 --2- 172.21.15.84:0/3762387444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f5510 0x7f38300f5930 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f382c004660 tx=0x7f382c009b30).stop 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 -- 172.21.15.84:0/3762387444 shutdown_connections 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 --2- 172.21.15.84:0/3762387444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f67b0 0x7f38300f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 --2- 172.21.15.84:0/3762387444 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f5510 0x7f38300f5930 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.130+0000 7f3836e0c700 1 -- 172.21.15.84:0/3762387444 >> 172.21.15.84:0/3762387444 conn(0x7f38300f0c70 msgr2=0x7f38300f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.131+0000 7f3836e0c700 1 -- 172.21.15.84:0/3762387444 shutdown_connections 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.131+0000 7f3836e0c700 1 -- 172.21.15.84:0/3762387444 wait complete. 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.131+0000 7f3836e0c700 1 Processor -- start 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.131+0000 7f3836e0c700 1 -- start start 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.131+0000 7f3836e0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f5510 0x7f3830109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3836e0c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:55.133 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3836e0c700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300609d0 con 0x7f38300f67b0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3836e0c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3830060b10 con 0x7f38300f5510 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:35842/0 (socket says 172.21.15.84:35842) 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 -- 172.21.15.84:0/1624533538 learned_addr learned my addr 172.21.15.84:0/1624533538 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835e0a700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f5510 0x7f3830109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 -- 172.21.15.84:0/1624533538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f5510 msgr2=0x7f3830109b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f5510 0x7f3830109b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 -- 172.21.15.84:0/1624533538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c005040 con 0x7f38300f67b0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3835609700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f382000b410 tx=0x7f3820006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38200125f0 con 0x7f38300f67b0 2024-01-18T07:38:55.134 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f382000be30 con 0x7f38300f67b0 2024-01-18T07:38:55.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383010c6f0 con 0x7f38300f67b0 2024-01-18T07:38:55.135 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3820002a20 con 0x7f38300f67b0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.132+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383010cc80 con 0x7f38300f67b0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.133+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3830045ad0 con 0x7f38300f67b0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.134+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f382000f030 con 0x7f38300f67b0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.134+0000 7f38267fc700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 0x7f381c05e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.134+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f38200762e0 con 0x7f38300f67b0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.134+0000 7f3835e0a700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 0x7f381c05e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:38:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.135+0000 7f3835e0a700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 0x7f381c05e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f38300605c0 tx=0x7f382c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:38:55.138 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.138+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38200493d0 con 0x7f38300f67b0 2024-01-18T07:38:55.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.345+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38300fde60 con 0x7f38300f67b0 2024-01-18T07:38:55.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.346+0000 7f38267fc700 1 -- 172.21.15.84:0/1624533538 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f382004c9f0 con 0x7f38300f67b0 2024-01-18T07:38:55.346 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 msgr2=0x7f381c05e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 0x7f381c05e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f38300605c0 tx=0x7f382c009b30).stop 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 msgr2=0x7f383010c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f382000b410 tx=0x7f3820006ef0).stop 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 shutdown_connections 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f381c05c2b0 0x7f381c05e770 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f38300f5510 0x7f3830109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 --2- 172.21.15.84:0/1624533538 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f38300f67b0 0x7f383010c090 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:38:55.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 >> 172.21.15.84:0/1624533538 conn(0x7f38300f0c70 msgr2=0x7f38300fa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:38:55.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 shutdown_connections 2024-01-18T07:38:55.349 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:38:55.347+0000 7f3836e0c700 1 -- 172.21.15.84:0/1624533538 wait complete. 2024-01-18T07:38:55.373 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:55 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:55.056383+0000 mgr.smithi084.sihbqg (mgr.14162) 645 : cluster [DBG] pgmap v606: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:55.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:55 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:55.056383+0000 mgr.smithi084.sihbqg (mgr.14162) 645 : cluster [DBG] pgmap v606: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:55.762 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:38:56.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:56 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:38:55.346685+0000 mon.smithi084 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.84:0/1624533538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:56.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:56 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:38:55.346685+0000 mon.smithi084 (mon.0) 572 : audit [DBG] from='client.? 172.21.15.84:0/1624533538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:38:57.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:57 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:57.056718+0000 mgr.smithi084.sihbqg (mgr.14162) 646 : cluster [DBG] pgmap v607: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:57.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:57 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:57.056718+0000 mgr.smithi084.sihbqg (mgr.14162) 646 : cluster [DBG] pgmap v607: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:58.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:38:58.997 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:38:59.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:38:59 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:38:59.057176+0000 mgr.smithi084.sihbqg (mgr.14162) 647 : cluster [DBG] pgmap v608: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:38:59.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:38:59 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:38:59.057176+0000 mgr.smithi084.sihbqg (mgr.14162) 647 : cluster [DBG] pgmap v608: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 -- 172.21.15.84:0/2457789998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 msgr2=0x7f4ed00f9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 --2- 172.21.15.84:0/2457789998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed00f9400 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4eb8004660 tx=0x7f4eb8009b30).stop 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 -- 172.21.15.84:0/2457789998 shutdown_connections 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 --2- 172.21.15.84:0/2457789998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed00fbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 --2- 172.21.15.84:0/2457789998 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed00f9400 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.069 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 -- 172.21.15.84:0/2457789998 >> 172.21.15.84:0/2457789998 conn(0x7f4ed00f0e20 msgr2=0x7f4ed00f3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 -- 172.21.15.84:0/2457789998 shutdown_connections 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 -- 172.21.15.84:0/2457789998 wait complete. 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.067+0000 7f4ed6110700 1 Processor -- start 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 -- start start 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed01075d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0107b80 con 0x7f4ed0060810 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed0107cc0 con 0x7f4ed00f9980 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34962/0 (socket says 172.21.15.84:34962) 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 -- 172.21.15.84:0/80064243 learned_addr learned my addr 172.21.15.84:0/80064243 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4eceffd700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed01075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 -- 172.21.15.84:0/80064243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 msgr2=0x7f4ed01075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:00.070 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed01075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 -- 172.21.15.84:0/80064243 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8005040 con 0x7f4ed0060810 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4eceffd700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed01075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ecf7fe700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4eb8009830 tx=0x7f4eb8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb8018070 con 0x7f4ed0060810 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4eb8013410 con 0x7f4ed0060810 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4eb801c620 con 0x7f4ed0060810 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0108600 con 0x7f4ed0060810 2024-01-18T07:39:00.071 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.068+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed0108b60 con 0x7f4ed0060810 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.069+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed0045ad0 con 0x7f4ed0060810 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.070+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f4eb801c890 con 0x7f4ed0060810 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.070+0000 7f4ec7fff700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 0x7f4ebc05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.070+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f4eb8077970 con 0x7f4ed0060810 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.070+0000 7f4eceffd700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 0x7f4ebc05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:00.072 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.070+0000 7f4eceffd700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 0x7f4ebc05e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4ec00041d0 tx=0x7f4ec0009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:00.074 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.074+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4eb804a990 con 0x7f4ed0060810 2024-01-18T07:39:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.284+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ed00ff0b0 con 0x7f4ed0060810 2024-01-18T07:39:00.285 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.284+0000 7f4ec7fff700 1 -- 172.21.15.84:0/80064243 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f4eb804dfb0 con 0x7f4ed0060810 2024-01-18T07:39:00.285 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:00.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 msgr2=0x7f4ebc05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:00.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 0x7f4ebc05e760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4ec00041d0 tx=0x7f4ec0009040).stop 2024-01-18T07:39:00.286 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 msgr2=0x7f4ed01070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4eb8009830 tx=0x7f4eb8009b30).stop 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 shutdown_connections 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f4ed0060810 0x7f4ed01070e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f4ebc05c2a0 0x7f4ebc05e760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 --2- 172.21.15.84:0/80064243 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ed00f9980 0x7f4ed01075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 >> 172.21.15.84:0/80064243 conn(0x7f4ed00f0e20 msgr2=0x7f4ed00f3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 shutdown_connections 2024-01-18T07:39:00.287 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:00.286+0000 7f4ed6110700 1 -- 172.21.15.84:0/80064243 wait complete. 2024-01-18T07:39:00.627 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:00 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:00.285294+0000 mon.smithi084 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.84:0/80064243' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:00.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:00 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:00.285294+0000 mon.smithi084 (mon.0) 573 : audit [DBG] from='client.? 172.21.15.84:0/80064243' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:00.693 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:01.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:01 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:01.057584+0000 mgr.smithi084.sihbqg (mgr.14162) 648 : cluster [DBG] pgmap v609: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:01.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:01 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:01.057584+0000 mgr.smithi084.sihbqg (mgr.14162) 648 : cluster [DBG] pgmap v609: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:03.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:03 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:03.058012+0000 mgr.smithi084.sihbqg (mgr.14162) 649 : cluster [DBG] pgmap v610: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:03.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:03 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:03.058012+0000 mgr.smithi084.sihbqg (mgr.14162) 649 : cluster [DBG] pgmap v610: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:03.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:03.929 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 -- 172.21.15.84:0/1393309859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 msgr2=0x7fced80f5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 --2- 172.21.15.84:0/1393309859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f5b00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fcec8004660 tx=0x7fcec8009b30).stop 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 -- 172.21.15.84:0/1393309859 shutdown_connections 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 --2- 172.21.15.84:0/1393309859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 0x7fced80f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 --2- 172.21.15.84:0/1393309859 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f5b00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 -- 172.21.15.84:0/1393309859 >> 172.21.15.84:0/1393309859 conn(0x7fced80f0e20 msgr2=0x7fced80f3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 -- 172.21.15.84:0/1393309859 shutdown_connections 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 -- 172.21.15.84:0/1393309859 wait complete. 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.994+0000 7fcedf627700 1 Processor -- start 2024-01-18T07:39:04.996 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 -- start start 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 0x7fced80f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced80f8220 con 0x7fced80f56e0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced80f8360 con 0x7fced80f6980 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34978/0 (socket says 172.21.15.84:34978) 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 -- 172.21.15.84:0/2896390249 learned_addr learned my addr 172.21.15.84:0/2896390249 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedcbc2700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 0x7fced80f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 -- 172.21.15.84:0/2896390249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 msgr2=0x7fced80f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 0x7fced80f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 -- 172.21.15.84:0/2896390249 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec8005040 con 0x7fced80f56e0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedd3c3700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fcec8004e30 tx=0x7fcec8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec8018070 con 0x7fced80f56e0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced80f85e0 con 0x7fced80f56e0 2024-01-18T07:39:04.997 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcec8013410 con 0x7fced80f56e0 2024-01-18T07:39:04.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.995+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec801c3c0 con 0x7fced80f56e0 2024-01-18T07:39:04.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.996+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced80f8b40 con 0x7fced80f56e0 2024-01-18T07:39:04.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.997+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced8045ad0 con 0x7fced80f56e0 2024-01-18T07:39:04.998 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.997+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fcec801d420 con 0x7fced80f56e0 2024-01-18T07:39:04.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.997+0000 7fcecdffb700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 0x7fcec405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:04.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.997+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fcec8077820 con 0x7fced80f56e0 2024-01-18T07:39:04.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.998+0000 7fcedcbc2700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 0x7fcec405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:04.999 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:04.998+0000 7fcedcbc2700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 0x7fcec405e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fced4009760 tx=0x7fced4009c60).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:05.002 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.001+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcec804a820 con 0x7fced80f56e0 2024-01-18T07:39:05.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.211+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fced80fde70 con 0x7fced80f56e0 2024-01-18T07:39:05.212 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.212+0000 7fcecdffb700 1 -- 172.21.15.84:0/2896390249 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fcec804de40 con 0x7fced80f56e0 2024-01-18T07:39:05.213 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 msgr2=0x7fcec405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 0x7fcec405e760 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fced4009760 tx=0x7fced4009c60).stop 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 msgr2=0x7fced80f9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fcec8004e30 tx=0x7fcec8009b30).stop 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 shutdown_connections 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fced80f56e0 0x7fced80f9840 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fcec405c2a0 0x7fcec405e760 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 --2- 172.21.15.84:0/2896390249 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fced80f6980 0x7fced80f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 >> 172.21.15.84:0/2896390249 conn(0x7fced80f0e20 msgr2=0x7fced80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:05.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 shutdown_connections 2024-01-18T07:39:05.215 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:05.213+0000 7fcedf627700 1 -- 172.21.15.84:0/2896390249 wait complete. 2024-01-18T07:39:05.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:05 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:05.058335+0000 mgr.smithi084.sihbqg (mgr.14162) 650 : cluster [DBG] pgmap v611: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:05.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:05 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:05.058335+0000 mgr.smithi084.sihbqg (mgr.14162) 650 : cluster [DBG] pgmap v611: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:05.694 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:06.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:06 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:05.212870+0000 mon.smithi084 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.84:0/2896390249' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:06.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:06 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:05.212870+0000 mon.smithi084 (mon.0) 574 : audit [DBG] from='client.? 172.21.15.84:0/2896390249' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:07.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:07 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:07.058760+0000 mgr.smithi084.sihbqg (mgr.14162) 651 : cluster [DBG] pgmap v612: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:07.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:07 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:07.058760+0000 mgr.smithi084.sihbqg (mgr.14162) 651 : cluster [DBG] pgmap v612: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:08.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:08.929 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:09.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:09 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:09.059026+0000 mgr.smithi084.sihbqg (mgr.14162) 652 : cluster [DBG] pgmap v613: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:09.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:09 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:09.059026+0000 mgr.smithi084.sihbqg (mgr.14162) 652 : cluster [DBG] pgmap v613: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.947+0000 7f82b759e700 1 -- 172.21.15.84:0/1465388121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 msgr2=0x7f82b80fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.947+0000 7f82b759e700 1 --2- 172.21.15.84:0/1465388121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80fbbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f82ac004660 tx=0x7f82ac009b30).stop 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 -- 172.21.15.84:0/1465388121 shutdown_connections 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 --2- 172.21.15.84:0/1465388121 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80fbbc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 --2- 172.21.15.84:0/1465388121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 0x7f82b80f91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 -- 172.21.15.84:0/1465388121 >> 172.21.15.84:0/1465388121 conn(0x7f82b80f0c70 msgr2=0x7f82b80f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:09.953 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 -- 172.21.15.84:0/1465388121 shutdown_connections 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 -- 172.21.15.84:0/1465388121 wait complete. 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.948+0000 7f82b759e700 1 Processor -- start 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 -- start start 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 0x7f82b80f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80f5650 con 0x7f82b80f9770 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80f5790 con 0x7f82b8060870 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34870/0 (socket says 172.21.15.84:34870) 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 -- 172.21.15.84:0/1568747174 learned_addr learned my addr 172.21.15.84:0/1568747174 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b659c700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 0x7f82b80f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 -- 172.21.15.84:0/1568747174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 msgr2=0x7f82b80f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 0x7f82b80f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 -- 172.21.15.84:0/1568747174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac005040 con 0x7f82b80f9770 2024-01-18T07:39:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b5d9b700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f82ac000c00 tx=0x7f82ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ac019070 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f82ac007e70 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b80f5a10 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82ac014d40 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.949+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b80f5f70 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.950+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8045ad0 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.951+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f82ac025070 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.951+0000 7f82a6ffd700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 0x7f82a005e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.951+0000 7f82b659c700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 0x7f82a005e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.951+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f82ac076b10 con 0x7f82b80f9770 2024-01-18T07:39:09.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.951+0000 7f82b659c700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 0x7f82a005e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f82b80f6a00 tx=0x7f82a800b040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:09.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:09.955+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82ac049b10 con 0x7f82b80f9770 2024-01-18T07:39:10.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.167+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82b8027a00 con 0x7f82b80f9770 2024-01-18T07:39:10.168 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.167+0000 7f82a6ffd700 1 -- 172.21.15.84:0/1568747174 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f82ac04d130 con 0x7f82b80f9770 2024-01-18T07:39:10.168 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 msgr2=0x7f82a005e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 0x7f82a005e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f82b80f6a00 tx=0x7f82a800b040).stop 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 msgr2=0x7f82b80f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f82ac000c00 tx=0x7f82ac00b040).stop 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 shutdown_connections 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f82a005c2b0 0x7f82a005e770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82b8060870 0x7f82b80f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 --2- 172.21.15.84:0/1568747174 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f82b80f9770 0x7f82b80f50f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 >> 172.21.15.84:0/1568747174 conn(0x7f82b80f0c70 msgr2=0x7f82b80f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:10.170 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 shutdown_connections 2024-01-18T07:39:10.171 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:10.169+0000 7f82b759e700 1 -- 172.21.15.84:0/1568747174 wait complete. 2024-01-18T07:39:10.561 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:10 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:10.168481+0000 mon.smithi084 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.84:0/1568747174' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:10.583 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:10.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:10 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:10.168481+0000 mon.smithi084 (mon.0) 575 : audit [DBG] from='client.? 172.21.15.84:0/1568747174' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:11.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:11 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:11.059314+0000 mgr.smithi084.sihbqg (mgr.14162) 653 : cluster [DBG] pgmap v614: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:11.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:11 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:11.059314+0000 mgr.smithi084.sihbqg (mgr.14162) 653 : cluster [DBG] pgmap v614: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:13.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:13 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:13.059676+0000 mgr.smithi084.sihbqg (mgr.14162) 654 : cluster [DBG] pgmap v615: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:13.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:13 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:13.059676+0000 mgr.smithi084.sihbqg (mgr.14162) 654 : cluster [DBG] pgmap v615: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:13.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:13.818 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 -- 172.21.15.84:0/2508375993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 msgr2=0x7f9ab80f9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2508375993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab80f9290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f9ab4004660 tx=0x7f9ab4009b30).stop 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 -- 172.21.15.84:0/2508375993 shutdown_connections 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2508375993 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 0x7f9ab80fbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2508375993 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab80f9290 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 -- 172.21.15.84:0/2508375993 >> 172.21.15.84:0/2508375993 conn(0x7f9ab80f0c70 msgr2=0x7f9ab80f3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 -- 172.21.15.84:0/2508375993 shutdown_connections 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.938+0000 7f9abe94f700 1 -- 172.21.15.84:0/2508375993 wait complete. 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 Processor -- start 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 -- start start 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:14.942 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 0x7f9ab81073e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8107990 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abe94f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab8107ad0 con 0x7f9ab80f9780 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abd94d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abd94d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:34888/0 (socket says 172.21.15.84:34888) 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.939+0000 7f9abd94d700 1 -- 172.21.15.84:0/2336063117 learned_addr learned my addr 172.21.15.84:0/2336063117 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abd14c700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 0x7f9ab81073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abd94d700 1 -- 172.21.15.84:0/2336063117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 msgr2=0x7f9ab81073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abd94d700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 0x7f9ab81073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abd94d700 1 -- 172.21.15.84:0/2336063117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4005040 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abd94d700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9ab4009830 tx=0x7f9ab4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab4018070 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ab4013410 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ab401c620 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab8108410 con 0x7f9ab80607d0 2024-01-18T07:39:14.943 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.940+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab8108970 con 0x7f9ab80607d0 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.941+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab8045ad0 con 0x7f9ab80607d0 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.941+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f9ab401d960 con 0x7f9ab80607d0 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.941+0000 7f9aae7fc700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 0x7f9aa405e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.941+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f9ab4077a30 con 0x7f9ab80607d0 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.942+0000 7f9abd14c700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 0x7f9aa405e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:14.944 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.942+0000 7f9abd14c700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 0x7f9aa405e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9aa80041d0 tx=0x7f9aa8009040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:14.946 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:14.945+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ab404aa50 con 0x7f9ab80607d0 2024-01-18T07:39:15.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.154+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9ab80fef10 con 0x7f9ab80607d0 2024-01-18T07:39:15.155 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.155+0000 7f9aae7fc700 1 -- 172.21.15.84:0/2336063117 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f9ab404e070 con 0x7f9ab80607d0 2024-01-18T07:39:15.155 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 msgr2=0x7f9aa405e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 0x7f9aa405e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f9aa80041d0 tx=0x7f9aa8009040).stop 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 msgr2=0x7f9ab8106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9ab4009830 tx=0x7f9ab4009b30).stop 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 shutdown_connections 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f9ab80607d0 0x7f9ab8106ef0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f9aa405c2b0 0x7f9aa405e770 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 --2- 172.21.15.84:0/2336063117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9ab80f9780 0x7f9ab81073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:15.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.156+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 >> 172.21.15.84:0/2336063117 conn(0x7f9ab80f0c70 msgr2=0x7f9ab80f3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:15.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.157+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 shutdown_connections 2024-01-18T07:39:15.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:15.157+0000 7f9abe94f700 1 -- 172.21.15.84:0/2336063117 wait complete. 2024-01-18T07:39:15.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:15 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:15.060017+0000 mgr.smithi084.sihbqg (mgr.14162) 655 : cluster [DBG] pgmap v616: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:15.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:15 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:15.060017+0000 mgr.smithi084.sihbqg (mgr.14162) 655 : cluster [DBG] pgmap v616: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:15.532 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:16.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:16 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:15.155818+0000 mon.smithi084 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.84:0/2336063117' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:16.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:16 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:15.155818+0000 mon.smithi084 (mon.0) 576 : audit [DBG] from='client.? 172.21.15.84:0/2336063117' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:17.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:17 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:17.060546+0000 mgr.smithi084.sihbqg (mgr.14162) 656 : cluster [DBG] pgmap v617: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:17.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:17 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:17.060546+0000 mgr.smithi084.sihbqg (mgr.14162) 656 : cluster [DBG] pgmap v617: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:18.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:18.767 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:19.368 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:19 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:19.060933+0000 mgr.smithi084.sihbqg (mgr.14162) 657 : cluster [DBG] pgmap v618: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:19.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:19 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:19.060933+0000 mgr.smithi084.sihbqg (mgr.14162) 657 : cluster [DBG] pgmap v618: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:19.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- 172.21.15.84:0/2059088288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f6970 msgr2=0x7f6ab80f6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:19.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/2059088288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f6dd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f6ab4004660 tx=0x7f6ab4009b30).stop 2024-01-18T07:39:19.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- 172.21.15.84:0/2059088288 shutdown_connections 2024-01-18T07:39:19.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/2059088288 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f6dd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:19.955 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/2059088288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- 172.21.15.84:0/2059088288 >> 172.21.15.84:0/2059088288 conn(0x7f6ab80f0e10 msgr2=0x7f6ab80f3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- 172.21.15.84:0/2059088288 shutdown_connections 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- 172.21.15.84:0/2059088288 wait complete. 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 Processor -- start 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.953+0000 7f6ac05b0700 1 -- start start 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80f82a0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80f83e0 con 0x7f6ab80f6970 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42596/0 (socket says 172.21.15.84:42596) 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 -- 172.21.15.84:0/1559515837 learned_addr learned my addr 172.21.15.84:0/1559515837 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abdb4b700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:19.956 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 -- 172.21.15.84:0/1559515837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f6970 msgr2=0x7f6ab80f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 -- 172.21.15.84:0/1559515837 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab4005040 con 0x7f6ab80f56d0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6abe34c700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f6aa80093f0 tx=0x7f6aa8006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aa8009ea0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6aa8013af0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6aa800dad0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab80f86c0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.957 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.954+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab810f130 con 0x7f6ab80f56d0 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.955+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8045ad0 con 0x7f6ab80f56d0 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.956+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f6aa801a060 con 0x7f6ab80f56d0 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.956+0000 7f6aaeffd700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 0x7f6aa405e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.956+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f6aa8075b00 con 0x7f6ab80f56d0 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.956+0000 7f6abdb4b700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 0x7f6aa405e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:19.958 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.956+0000 7f6abdb4b700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 0x7f6aa405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6ab4013580 tx=0x7f6ab4009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:19.960 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:19.960+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6aa8048b20 con 0x7f6ab80f56d0 2024-01-18T07:39:20.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.180+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6ab80fdf00 con 0x7f6ab80f56d0 2024-01-18T07:39:20.181 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.180+0000 7f6aaeffd700 1 -- 172.21.15.84:0/1559515837 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f6aa804c140 con 0x7f6ab80f56d0 2024-01-18T07:39:20.182 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:20.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 msgr2=0x7f6aa405e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:20.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 0x7f6aa405e760 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6ab4013580 tx=0x7f6ab4009b30).stop 2024-01-18T07:39:20.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 msgr2=0x7f6ab80f9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:20.183 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f6aa80093f0 tx=0x7f6aa8006ec0).stop 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 shutdown_connections 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f6ab80f56d0 0x7f6ab80f9830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f6aa405c2a0 0x7f6aa405e760 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 --2- 172.21.15.84:0/1559515837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ab80f6970 0x7f6ab80f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 >> 172.21.15.84:0/1559515837 conn(0x7f6ab80f0e10 msgr2=0x7f6ab80603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 shutdown_connections 2024-01-18T07:39:20.184 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:20.182+0000 7f6ac05b0700 1 -- 172.21.15.84:0/1559515837 wait complete. 2024-01-18T07:39:20.538 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:20 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:20.181568+0000 mon.smithi084 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.84:0/1559515837' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:20.563 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:20.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:20 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:20.181568+0000 mon.smithi084 (mon.0) 577 : audit [DBG] from='client.? 172.21.15.84:0/1559515837' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:21.632 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:21 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:21.061329+0000 mgr.smithi084.sihbqg (mgr.14162) 658 : cluster [DBG] pgmap v619: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:21.664 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:21 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:21.061329+0000 mgr.smithi084.sihbqg (mgr.14162) 658 : cluster [DBG] pgmap v619: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:23.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:23 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:23.061874+0000 mgr.smithi084.sihbqg (mgr.14162) 659 : cluster [DBG] pgmap v620: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:23.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:23 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:23.061874+0000 mgr.smithi084.sihbqg (mgr.14162) 659 : cluster [DBG] pgmap v620: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:23.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:23.799 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.776+0000 7fdc55656700 1 -- 172.21.15.84:0/3024966029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 msgr2=0x7fdc500f6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.776+0000 7fdc55656700 1 --2- 172.21.15.84:0/3024966029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc500f6c20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fdc40004660 tx=0x7fdc40009b30).stop 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 -- 172.21.15.84:0/3024966029 shutdown_connections 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 --2- 172.21.15.84:0/3024966029 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc500f6c20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 --2- 172.21.15.84:0/3024966029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc500f5520 0x7fdc500f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 -- 172.21.15.84:0/3024966029 >> 172.21.15.84:0/3024966029 conn(0x7fdc500f0ca0 msgr2=0x7fdc500f30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 -- 172.21.15.84:0/3024966029 shutdown_connections 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 -- 172.21.15.84:0/3024966029 wait complete. 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 Processor -- start 2024-01-18T07:39:24.780 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.777+0000 7fdc55656700 1 -- start start 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc500f5520 0x7fdc50109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50060940 con 0x7fdc500f67c0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc50060a80 con 0x7fdc500f5520 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:42614/0 (socket says 172.21.15.84:42614) 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 -- 172.21.15.84:0/892326715 learned_addr learned my addr 172.21.15.84:0/892326715 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 -- 172.21.15.84:0/892326715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc500f5520 msgr2=0x7fdc50109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc500f5520 0x7fdc50109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 -- 172.21.15.84:0/892326715 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40005040 con 0x7fdc500f67c0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4f7fe700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fdc40009f50 tx=0x7fdc40009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc40018070 con 0x7fdc500f67c0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc40013410 con 0x7fdc500f67c0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc5010c630 con 0x7fdc500f67c0 2024-01-18T07:39:24.781 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc4001c3c0 con 0x7fdc500f67c0 2024-01-18T07:39:24.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.778+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5010cb90 con 0x7fdc500f67c0 2024-01-18T07:39:24.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.779+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50045ad0 con 0x7fdc500f67c0 2024-01-18T07:39:24.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.780+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7fdc4001d420 con 0x7fdc500f67c0 2024-01-18T07:39:24.782 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.780+0000 7fdc4cff9700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 0x7fdc3805e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:24.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.780+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7fdc40077860 con 0x7fdc500f67c0 2024-01-18T07:39:24.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.780+0000 7fdc4ffff700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 0x7fdc3805e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:24.783 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.780+0000 7fdc4ffff700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 0x7fdc3805e770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdc500605c0 tx=0x7fdc440099e0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:24.785 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.784+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc4004a860 con 0x7fdc500f67c0 2024-01-18T07:39:24.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.990+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc50027a00 con 0x7fdc500f67c0 2024-01-18T07:39:24.992 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.990+0000 7fdc4cff9700 1 -- 172.21.15.84:0/892326715 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7fdc4004de80 con 0x7fdc500f67c0 2024-01-18T07:39:24.992 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 msgr2=0x7fdc3805e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 0x7fdc3805e770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdc500605c0 tx=0x7fdc440099e0).stop 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 msgr2=0x7fdc5010c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7fdc40009f50 tx=0x7fdc40009b30).stop 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 shutdown_connections 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7fdc3805c2b0 0x7fdc3805e770 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdc500f5520 0x7fdc50109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 --2- 172.21.15.84:0/892326715 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7fdc500f67c0 0x7fdc5010c0e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:24.994 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 >> 172.21.15.84:0/892326715 conn(0x7fdc500f0ca0 msgr2=0x7fdc500fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 shutdown_connections 2024-01-18T07:39:24.995 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:24.992+0000 7fdc55656700 1 -- 172.21.15.84:0/892326715 wait complete. 2024-01-18T07:39:25.345 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:25 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:24.992610+0000 mon.smithi084 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.84:0/892326715' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:25.412 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:25.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:25 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:24.992610+0000 mon.smithi084 (mon.0) 578 : audit [DBG] from='client.? 172.21.15.84:0/892326715' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:26.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:26 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:25.062204+0000 mgr.smithi084.sihbqg (mgr.14162) 660 : cluster [DBG] pgmap v621: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:26.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:26 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:25.062204+0000 mgr.smithi084.sihbqg (mgr.14162) 660 : cluster [DBG] pgmap v621: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:27.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:27 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:27.062579+0000 mgr.smithi084.sihbqg (mgr.14162) 661 : cluster [DBG] pgmap v622: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:27.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:27 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:27.062579+0000 mgr.smithi084.sihbqg (mgr.14162) 661 : cluster [DBG] pgmap v622: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:28.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:28.648 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:29.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:29 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:29.062938+0000 mgr.smithi084.sihbqg (mgr.14162) 662 : cluster [DBG] pgmap v623: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:29.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:29 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:29.062938+0000 mgr.smithi084.sihbqg (mgr.14162) 662 : cluster [DBG] pgmap v623: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:29.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 -- 172.21.15.84:0/2749622039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f7a70 msgr2=0x7f787c0f7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:29.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2749622039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f7a70 0x7f787c0f7ed0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f786c004660 tx=0x7f786c009b30).stop 2024-01-18T07:39:29.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 -- 172.21.15.84:0/2749622039 shutdown_connections 2024-01-18T07:39:29.716 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2749622039 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f7a70 0x7f787c0f7ed0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2749622039 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f67d0 0x7f787c0f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 -- 172.21.15.84:0/2749622039 >> 172.21.15.84:0/2749622039 conn(0x7f787c0f1f50 msgr2=0x7f787c0f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 -- 172.21.15.84:0/2749622039 shutdown_connections 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.712+0000 7f7883eb4700 1 -- 172.21.15.84:0/2749622039 wait complete. 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 Processor -- start 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 -- start start 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f7a70 0x7f787c1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c107bf0 con 0x7f787c0f67d0 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7883eb4700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787c1084a0 con 0x7f787c0f7a70 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7881c50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7881c50700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59572/0 (socket says 172.21.15.84:59572) 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f7881c50700 1 -- 172.21.15.84:0/2393767191 learned_addr learned my addr 172.21.15.84:0/2393767191 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:29.717 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.713+0000 7f788144f700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f7a70 0x7f787c1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7881c50700 1 -- 172.21.15.84:0/2393767191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f7a70 msgr2=0x7f787c1075b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7881c50700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f7a70 0x7f787c1075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7881c50700 1 -- 172.21.15.84:0/2393767191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c005040 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7881c50700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f78780093f0 tx=0x7f7878006ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7878009ea0 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7878013af0 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787c108780 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787800dad0 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.714+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787c108d40 con 0x7f787c0f67d0 2024-01-18T07:39:29.718 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.715+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787c059100 con 0x7f787c0f67d0 2024-01-18T07:39:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.715+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f787801a060 con 0x7f787c0f67d0 2024-01-18T07:39:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.715+0000 7f78727fc700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 0x7f786805e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.716+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f7878075b40 con 0x7f787c0f67d0 2024-01-18T07:39:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.716+0000 7f788144f700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 0x7f786805e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:29.719 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.716+0000 7f788144f700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 0x7f786805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f786c009d60 tx=0x7f786c009b30).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:29.721 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.720+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7878048b60 con 0x7f787c0f67d0 2024-01-18T07:39:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.929+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f787c0ff0a0 con 0x7f787c0f67d0 2024-01-18T07:39:29.931 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.929+0000 7f78727fc700 1 -- 172.21.15.84:0/2393767191 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f787804c180 con 0x7f787c0f67d0 2024-01-18T07:39:29.931 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:29.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 msgr2=0x7f786805e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:29.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 0x7f786805e760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f786c009d60 tx=0x7f786c009b30).stop 2024-01-18T07:39:29.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 msgr2=0x7f787c1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:29.932 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f78780093f0 tx=0x7f7878006ec0).stop 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 shutdown_connections 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f787c0f67d0 0x7f787c1070c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f786805c2a0 0x7f786805e760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 --2- 172.21.15.84:0/2393767191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f787c0f7a70 0x7f787c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 >> 172.21.15.84:0/2393767191 conn(0x7f787c0f1f50 msgr2=0x7f787c0fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 shutdown_connections 2024-01-18T07:39:29.933 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:29.931+0000 7f7883eb4700 1 -- 172.21.15.84:0/2393767191 wait complete. 2024-01-18T07:39:30.378 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:30 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:29.931346+0000 mon.smithi084 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.84:0/2393767191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:30.397 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:30.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:30 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:29.931346+0000 mon.smithi084 (mon.0) 579 : audit [DBG] from='client.? 172.21.15.84:0/2393767191' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:31.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:31 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:31.063365+0000 mgr.smithi084.sihbqg (mgr.14162) 663 : cluster [DBG] pgmap v624: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:31.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:31 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:31.063365+0000 mgr.smithi084.sihbqg (mgr.14162) 663 : cluster [DBG] pgmap v624: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:33.382 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:33 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: cluster 2024-01-18T07:39:33.063791+0000 mgr.smithi084.sihbqg (mgr.14162) 664 : cluster [DBG] pgmap v625: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:33.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 -- ceph health --format=json 2024-01-18T07:39:33.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:33 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: cluster 2024-01-18T07:39:33.063791+0000 mgr.smithi084.sihbqg (mgr.14162) 664 : cluster [DBG] pgmap v625: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2024-01-18T07:39:33.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/mon.smithi084/config 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 -- 172.21.15.84:0/1550912657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 msgr2=0x7f15700f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 --2- 172.21.15.84:0/1550912657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f15700f6bf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1560004660 tx=0x7f1560009b30).stop 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 -- 172.21.15.84:0/1550912657 shutdown_connections 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 --2- 172.21.15.84:0/1550912657 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f15700f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 --2- 172.21.15.84:0/1550912657 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f15700f6bf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 -- 172.21.15.84:0/1550912657 >> 172.21.15.84:0/1550912657 conn(0x7f15700f1f50 msgr2=0x7f15700f4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 -- 172.21.15.84:0/1550912657 shutdown_connections 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.685+0000 7f157807d700 1 -- 172.21.15.84:0/1550912657 wait complete. 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 Processor -- start 2024-01-18T07:39:34.689 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 -- start start 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f1570069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 -- --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157006a410 con 0x7f15700f67d0 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f157807d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157006a550 con 0x7f15700f7a70 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 --2- >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.84:3300/0 says I am v2:172.21.15.84:59594/0 (socket says 172.21.15.84:59594) 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 -- 172.21.15.84:0/237970071 learned_addr learned my addr 172.21.15.84:0/237970071 (peer_addr_for_me v2:172.21.15.84:0/0) 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 -- 172.21.15.84:0/237970071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 msgr2=0x7f1570069e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575618700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f1570069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f1570069e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 -- 172.21.15.84:0/237970071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560005040 con 0x7f15700f67d0 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575618700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f1570069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2024-01-18T07:39:34.690 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.686+0000 7f1575e19700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1560009f50 tx=0x7f1560009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1560018070 con 0x7f15700f67d0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1560013410 con 0x7f15700f67d0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157006a7d0 con 0x7f15700f67d0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f156001c620 con 0x7f15700f67d0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157010f110 con 0x7f15700f67d0 2024-01-18T07:39:34.691 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.687+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570045ad0 con 0x7f15700f67d0 2024-01-18T07:39:34.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.688+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 4 ==== mgrmap(e 18) v1 ==== 72193+0+0 (secure 0 0 0) 0x7f156001d420 con 0x7f15700f67d0 2024-01-18T07:39:34.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.688+0000 7f15667fc700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 0x7f155c05e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-18T07:39:34.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.688+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4552+0+0 (secure 0 0 0) 0x7f15600779f0 con 0x7f15700f67d0 2024-01-18T07:39:34.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.688+0000 7f1575618700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 0x7f155c05e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-18T07:39:34.692 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.689+0000 7f1575618700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 0x7f155c05e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f156c00a880 tx=0x7f156c008040).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-18T07:39:34.693 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.692+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1560079e60 con 0x7f15700f67d0 2024-01-18T07:39:34.902 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.900+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 --> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1570027a00 con 0x7f15700f67d0 2024-01-18T07:39:34.903 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.900+0000 7f15667fc700 1 -- 172.21.15.84:0/237970071 <== mon.0 v2:172.21.15.84:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+172 (secure 0 0 0) 0x7f156004a9f0 con 0x7f15700f67d0 2024-01-18T07:39:34.903 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 msgr2=0x7f155c05e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 0x7f155c05e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f156c00a880 tx=0x7f156c008040).stop 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 msgr2=0x7f157006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1560009f50 tx=0x7f1560009b30).stop 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 shutdown_connections 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] conn(0x7f15700f67d0 0x7f157006b9a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.84:6800/2258764086,v1:172.21.15.84:6801/2258764086] conn(0x7f155c05c2a0 0x7f155c05e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 --2- 172.21.15.84:0/237970071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15700f7a70 0x7f1570069e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-18T07:39:34.904 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 >> 172.21.15.84:0/237970071 conn(0x7f15700f1f50 msgr2=0x7f15700fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-18T07:39:34.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 shutdown_connections 2024-01-18T07:39:34.905 INFO:teuthology.orchestra.run.smithi084.stderr:2024-01-18T07:39:34.902+0000 7f157807d700 1 -- 172.21.15.84:0/237970071 wait complete. 2024-01-18T07:39:35.266 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:34 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: audit 2024-01-18T07:39:34.902616+0000 mon.smithi084 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.84:0/237970071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:35.304 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_WARN","checks":{"CEPHADM_FAILED_DAEMON":{"severity":"HEALTH_WARN","summary":{"message":"1 failed cephadm daemon(s)","count":1},"muted":false}},"mutes":[]} 2024-01-18T07:39:35.304 INFO:tasks.cephadm:Teardown begin 2024-01-18T07:39:35.304 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-18T07:39:35.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:39:35.335 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:39:35.370 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:34 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: audit 2024-01-18T07:39:34.902616+0000 mon.smithi084 (mon.0) 580 : audit [DBG] from='client.? 172.21.15.84:0/237970071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-01-18T07:39:35.375 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-18T07:39:35.375 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-18T07:39:35.394 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-18T07:39:35.435 INFO:tasks.cephadm:Stopping all daemons... 2024-01-18T07:39:35.435 INFO:tasks.cephadm.mon.smithi084:Stopping mon.smithi084... 2024-01-18T07:39:35.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084 2024-01-18T07:39:35.587 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:35 smithi084 systemd[1]: Stopping Ceph mon.smithi084 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:35.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:39:35.585+0000 7fbf43606700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi084 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-18T07:39:35.876 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Jan 18 07:39:35 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi084[38736]: debug 2024-01-18T07:39:35.585+0000 7fbf43606700 -1 mon.smithi084@0(leader) e2 *** Got Signal Terminated *** 2024-01-18T07:39:36.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi084.service' 2024-01-18T07:39:36.098 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:36.098 INFO:tasks.cephadm.mon.smithi084:Stopped mon.smithi084 2024-01-18T07:39:36.098 INFO:tasks.cephadm.mon.smithi106:Stopping mon.smithi106... 2024-01-18T07:39:36.099 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi106 2024-01-18T07:39:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:36 smithi106 systemd[1]: Stopping Ceph mon.smithi106 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:39:36.224+0000 7fa270812700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi106 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-18T07:39:36.414 INFO:journalctl@ceph.mon.smithi106.smithi106.stdout:Jan 18 07:39:36 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-mon.smithi106[53101]: debug 2024-01-18T07:39:36.224+0000 7fa270812700 -1 mon.smithi106@1(peon) e2 *** Got Signal Terminated *** 2024-01-18T07:39:36.813 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@mon.smithi106.service' 2024-01-18T07:39:36.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:36.863 INFO:tasks.cephadm.mon.smithi106:Stopped mon.smithi106 2024-01-18T07:39:36.863 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-01-18T07:39:36.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.0 2024-01-18T07:39:37.148 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:39:36 smithi084 systemd[1]: Stopping Ceph osd.0 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:37.148 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:39:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:39:36.980+0000 7f9649f46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:37.148 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:39:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:39:36.980+0000 7f9649f46700 -1 osd.0 37 *** Got signal Terminated *** 2024-01-18T07:39:37.149 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:39:36 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0[48011]: debug 2024-01-18T07:39:36.980+0000 7f9649f46700 -1 osd.0 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:37.435 INFO:journalctl@ceph.osd.0.smithi084.stdout:Jan 18 07:39:37 smithi084 podman[75276]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.0 2024-01-18T07:39:38.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.0.service' 2024-01-18T07:39:38.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:38.161 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-01-18T07:39:38.162 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-01-18T07:39:38.162 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.1 2024-01-18T07:39:38.382 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:39:38 smithi084 systemd[1]: Stopping Ceph osd.1 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:38.383 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:39:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:39:38.292+0000 7f8e6b16a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:38.383 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:39:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:39:38.292+0000 7f8e6b16a700 -1 osd.1 37 *** Got signal Terminated *** 2024-01-18T07:39:38.383 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:39:38 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1[51151]: debug 2024-01-18T07:39:38.292+0000 7f8e6b16a700 -1 osd.1 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:38.767 INFO:journalctl@ceph.osd.1.smithi084.stdout:Jan 18 07:39:38 smithi084 podman[75456]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.1 2024-01-18T07:39:39.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.1.service' 2024-01-18T07:39:39.494 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:39.495 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-01-18T07:39:39.495 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-01-18T07:39:39.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.2 2024-01-18T07:39:39.772 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:39:39 smithi084 systemd[1]: Stopping Ceph osd.2 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:39.772 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:39:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:39:39.629+0000 7f951b7d1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:39.772 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:39:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:39:39.629+0000 7f951b7d1700 -1 osd.2 37 *** Got signal Terminated *** 2024-01-18T07:39:39.773 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:39:39 smithi084 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2[54304]: debug 2024-01-18T07:39:39.629+0000 7f951b7d1700 -1 osd.2 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:40.058 INFO:journalctl@ceph.osd.2.smithi084.stdout:Jan 18 07:39:39 smithi084 podman[75638]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.2 2024-01-18T07:39:40.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.2.service' 2024-01-18T07:39:40.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:40.822 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-01-18T07:39:40.822 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-01-18T07:39:40.822 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.3 2024-01-18T07:39:41.141 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:39:40 smithi106 systemd[1]: Stopping Ceph osd.3 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:41.141 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:39:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:39:40.980+0000 7fcbda960700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:41.141 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:39:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:39:40.980+0000 7fcbda960700 -1 osd.3 37 *** Got signal Terminated *** 2024-01-18T07:39:41.141 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:39:40 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3[56340]: debug 2024-01-18T07:39:40.980+0000 7fcbda960700 -1 osd.3 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:41.414 INFO:journalctl@ceph.osd.3.smithi106.stdout:Jan 18 07:39:41 smithi106 podman[65686]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.3 2024-01-18T07:39:42.189 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.3.service' 2024-01-18T07:39:42.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:42.229 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-01-18T07:39:42.229 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-01-18T07:39:42.229 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.4 2024-01-18T07:39:42.524 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:39:42 smithi106 systemd[1]: Stopping Ceph osd.4 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:42.524 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:39:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:39:42.380+0000 7fc04cc1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:42.524 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:39:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:39:42.380+0000 7fc04cc1f700 -1 osd.4 37 *** Got signal Terminated *** 2024-01-18T07:39:42.524 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:39:42 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4[59793]: debug 2024-01-18T07:39:42.380+0000 7fc04cc1f700 -1 osd.4 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:42.914 INFO:journalctl@ceph.osd.4.smithi106.stdout:Jan 18 07:39:42 smithi106 podman[65871]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.4 2024-01-18T07:39:43.688 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.4.service' 2024-01-18T07:39:43.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:43.728 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-01-18T07:39:43.728 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-01-18T07:39:43.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.5 2024-01-18T07:39:44.024 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:39:43 smithi106 systemd[1]: Stopping Ceph osd.5 for bf9448d2-b5d1-11ee-95ae-87774f69a715... 2024-01-18T07:39:44.024 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:39:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:39:43.880+0000 7fe0703ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-01-18T07:39:44.024 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:39:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:39:43.880+0000 7fe0703ad700 -1 osd.5 37 *** Got signal Terminated *** 2024-01-18T07:39:44.024 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:39:43 smithi106 ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5[63096]: debug 2024-01-18T07:39:43.880+0000 7fe0703ad700 -1 osd.5 37 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-18T07:39:44.394 INFO:journalctl@ceph.osd.5.smithi106.stdout:Jan 18 07:39:44 smithi106 podman[66054]: ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715-osd.5 2024-01-18T07:39:45.080 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bf9448d2-b5d1-11ee-95ae-87774f69a715@osd.5.service' 2024-01-18T07:39:45.119 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-18T07:39:45.120 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-01-18T07:39:45.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --force --keep-logs 2024-01-18T07:39:48.468 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --force --keep-logs 2024-01-18T07:40:57.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:40:57.321 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-18T07:40:57.351 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-18T07:40:57.353 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/ubuntu@smithi084.front.sepia.ceph.com/crash 2024-01-18T07:40:57.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash -- . 2024-01-18T07:40:57.409 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash: Cannot open: No such file or directory 2024-01-18T07:40:57.409 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-01-18T07:40:57.411 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/ubuntu@smithi106.front.sepia.ceph.com/crash 2024-01-18T07:40:57.412 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash -- . 2024-01-18T07:40:57.451 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/crash: Cannot open: No such file or directory 2024-01-18T07:40:57.451 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-01-18T07:40:57.452 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-18T07:40:57.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-18T07:40:57.483 INFO:teuthology.orchestra.run.smithi084.stderr:grep: /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph.log: No such file or directory 2024-01-18T07:40:57.485 INFO:tasks.cephadm:Compressing logs... 2024-01-18T07:40:57.485 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-18T07:40:57.528 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi084.stderr:: No such file or directory 2024-01-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log 2024-01-18T07:40:57.554 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.2.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:: No such file or directory 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi106.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.3.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi106.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.4.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi106.yhaxbc.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi084.sihbqg.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.0.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi084.log 2024-01-18T07:40:57.557 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi084.sihbqg.log: /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.0.log: /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi084.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi084.stderr: 90.4% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log.gz 2024-01-18T07:40:57.558 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.1.log 2024-01-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.4.log: /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi106.yhaxbc.log: 89.9% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-volume.log.gz 2024-01-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi106.stderr: 95.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-18T07:40:57.559 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.5.log 2024-01-18T07:40:57.560 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.5.log: 92.9% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi106.yhaxbc.log.gz 2024-01-18T07:40:57.592 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.4.log.gz 2024-01-18T07:40:57.600 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.5.log.gz 2024-01-18T07:40:57.609 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.1.log: 93.0% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.2.log.gz 2024-01-18T07:40:57.612 INFO:teuthology.orchestra.run.smithi084.stderr: 92.6% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.0.log.gz 2024-01-18T07:40:57.619 INFO:teuthology.orchestra.run.smithi106.stderr: 92.5% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.3.log.gz 2024-01-18T07:40:57.620 INFO:teuthology.orchestra.run.smithi084.stderr: 92.6% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-osd.1.log.gz 2024-01-18T07:40:57.673 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi106.log.gz 2024-01-18T07:40:57.674 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-01-18T07:40:57.675 INFO:teuthology.orchestra.run.smithi106.stderr:real 0m0.130s 2024-01-18T07:40:57.675 INFO:teuthology.orchestra.run.smithi106.stderr:user 0m0.269s 2024-01-18T07:40:57.675 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m0.026s 2024-01-18T07:40:57.766 INFO:teuthology.orchestra.run.smithi084.stderr: 93.2% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mgr.smithi084.sihbqg.log.gz 2024-01-18T07:40:57.957 INFO:teuthology.orchestra.run.smithi084.stderr: 90.4% -- replaced with /var/log/ceph/bf9448d2-b5d1-11ee-95ae-87774f69a715/ceph-mon.smithi084.log.gz 2024-01-18T07:40:57.958 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-01-18T07:40:57.958 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m0.415s 2024-01-18T07:40:57.958 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m0.769s 2024-01-18T07:40:57.958 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.046s 2024-01-18T07:40:57.959 INFO:tasks.cephadm:Archiving logs... 2024-01-18T07:40:57.959 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/ubuntu@smithi084.front.sepia.ceph.com/log 2024-01-18T07:40:57.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-18T07:40:58.170 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/ubuntu@smithi106.front.sepia.ceph.com/log 2024-01-18T07:40:58.171 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-18T07:40:58.253 INFO:tasks.cephadm:Removing cluster... 2024-01-18T07:40:58.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --force 2024-01-18T07:40:58.548 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bf9448d2-b5d1-11ee-95ae-87774f69a715 --force 2024-01-18T07:40:58.843 INFO:tasks.cephadm:Removing cephadm ... 2024-01-18T07:40:58.843 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-18T07:40:58.861 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-18T07:40:58.879 INFO:tasks.cephadm:Teardown complete 2024-01-18T07:40:58.879 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-18T07:40:59.091 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=861659351234475095e27b36a8de54c5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/cephadm.py", line 1502, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph.py", line 1482, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_491b5e13e5ce87708e2d21a24a4fb19ac79ef4f0/qa/tasks/ceph_manager.py", line 3184, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-01-18T07:40:59.094 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-18T07:40:59.104 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-18T07:40:59.112 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-18T07:40:59.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-18T07:40:59.134 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-18T07:40:59.211 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-18T07:40:59.220 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-18T07:40:59.221 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T07:40:59.223 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T07:40:59.238 INFO:teuthology.orchestra.run.smithi084.stderr:bash: ntpq: command not found 2024-01-18T07:40:59.240 INFO:teuthology.orchestra.run.smithi106.stderr:bash: ntpq: command not found 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +204us[ +204us] +/- 82ms 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 60 -1997us[-1970us] +/- 68ms 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +2226us[+2226us] +/- 84ms 2024-01-18T07:40:59.242 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:40:59.244 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T07:40:59.244 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-01-18T07:40:59.244 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +224us[ +270us] +/- 82ms 2024-01-18T07:40:59.244 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1974us[-1927us] +/- 68ms 2024-01-18T07:40:59.245 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +2250us[+2297us] +/- 84ms 2024-01-18T07:40:59.245 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T07:40:59.245 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-18T07:40:59.342 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-18T07:40:59.343 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-18T07:40:59.351 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-18T07:40:59.412 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-18T07:40:59.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T07:40:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:40:59.502 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T07:40:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:40:59.532 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-18T07:40:59.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-18T07:40:59.550 INFO:teuthology.task.internal:Duration was 1887.867742 seconds 2024-01-18T07:40:59.551 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-18T07:40:59.562 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-18T07:40:59.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-18T07:40:59.565 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-18T07:40:59.608 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T07:40:59.617 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T07:40:59.927 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-18T07:40:59.927 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-01-18T07:40:59.928 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-18T07:40:59.951 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-01-18T07:40:59.951 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-18T07:40:59.972 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-18T07:40:59.973 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-18T07:40:59.994 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-18T07:41:00.064 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-18T07:41:00.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-18T07:41:00.066 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-18T07:41:00.165 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-18T07:41:00.175 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-18T07:41:00.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-18T07:41:00.202 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-18T07:41:00.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-18T07:41:00.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-18T07:41:00.245 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-18T07:41:00.269 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-01-18T07:41:00.297 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-01-18T07:41:00.317 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-18T07:41:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:41:00.357 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-18T07:41:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T07:41:00.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-18T07:41:00.386 INFO:teuthology.task.internal:Transferring archived files... 2024-01-18T07:41:00.387 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/smithi084 2024-01-18T07:41:00.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-18T07:41:00.438 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659/remote/smithi106 2024-01-18T07:41:00.439 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-18T07:41:00.478 INFO:teuthology.task.internal:Removing archive directory... 2024-01-18T07:41:00.479 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-18T07:41:00.481 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-18T07:41:00.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-18T07:41:00.543 INFO:teuthology.task.internal:Not uploading archives. 2024-01-18T07:41:00.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-18T07:41:00.555 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-18T07:41:00.555 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-18T07:41:00.557 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-18T07:41:00.573 INFO:teuthology.orchestra.run.smithi084.stdout: 526072 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 18 07:41 /home/ubuntu/cephtest 2024-01-18T07:41:00.587 INFO:teuthology.orchestra.run.smithi106.stdout: 525353 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 18 07:41 /home/ubuntu/cephtest 2024-01-18T07:41:00.589 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-18T07:41:00.605 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-18T07:41:00.625 INFO:teuthology.nuke:Checking targets against current locks 2024-01-18T07:41:00.649 DEBUG:teuthology.nuke:shortname: smithi084 2024-01-18T07:41:00.650 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T07:41:00.673 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 07:01:32.459685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlGicQZ8bRPfx7Psd5HuE7IMf7D2XHgU74GViHlSAf+oFgpasZyB7V1umOdDbm/UO3j9E8MVISyJbkkSrrLx+c='} 2024-01-18T07:41:00.677 DEBUG:teuthology.nuke:shortname: smithi106 2024-01-18T07:41:00.677 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T07:41:00.704 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-17_22:43:41-rados-wip-yuri10-testing-2024-01-17-0759-pacific-distro-default-smithi/7519659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 07:01:32.461427', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwbKpEW+Zr3p+nSrCGIYNgaAS97TE1Zn82PniN69HLG85jkDjGrf1sxJcqwdu6DbvlduVBHq8PTELL3S0uwYD8='} 2024-01-18T07:41:00.729 INFO:teuthology.orchestra.console.smithi084:Power off 2024-01-18T07:41:00.729 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-18T07:41:00.750 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-01-18T07:41:00.751 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:00.772 INFO:teuthology.orchestra.console.smithi106:Power off 2024-01-18T07:41:00.773 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-18T07:41:00.784 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-01-18T07:41:00.797 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-01-18T07:41:00.797 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:00.819 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-18T07:41:04.785 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:04.821 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:04.896 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-01-18T07:41:04.933 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-18T07:41:08.902 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:08.934 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:09.012 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-01-18T07:41:09.043 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-01-18T07:41:13.016 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:13.044 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-18T07:41:13.126 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-01-18T07:41:13.153 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-01-18T07:41:13.228 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-01-18T07:41:13.254 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-01-18T07:41:13.359 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1887.867742061615 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=861659351234475095e27b36a8de54c5 status: fail success: false 2024-01-18T07:41:13.360 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T07:41:13.450 INFO:teuthology.run:FAIL